The best would be when the bud is small, but the vast majority of energy saved comes from preventing the fruit from developing. Energy is roughly equivalent to mass. If it is an indoor tree, you might want to enjoy the fragrance before you remove the blooms, you would still save at least 95% of the energy that would go into the fruiting process.
_________________
Skeet