Well,
difficult.
I simply let the fruit attached to the tree from flower to fruit for about 8 to 10 month. depending on the gowth and how it looks.
As my fruit to not get any cold, I have to harvest my fruit nearly completely green:
As you can see, the peel exhibits only a light yellow blush, but won't get any orange coloration, because for orange coloration a cold periode is need, just for the color breakdown in which cold brings the tree to remove the green chloroplast elements and replace it for with orange/red/yellow chromoplast elements. And those elements will be responsible for the orange peel color.
So if you fruit has a proper size, is long enough on the tree,
and cold was missing, well, maybe it's time to pick the fruits.
But there is no thing to making a head on flower and fruits at the same time on a tree. The tree will usually only keep and mature the number of fruit, which it can bear...