My tree is about three feet high from the soil and a week ago it had new leaves growing as well as two male flowers and two females that were forming.
Now some leaves are yellowing and falling off. Also, one male flower had opened and it smelled great, it fell apart after two days as well as the other, the females were on the other side of the tree and when one was looking like it might open up it fell off. The other dried up and the new leaves curled in on themselves.

Could this be caused by those little black fruit/fungus flies? I'm currently spraying every 3-4 days with neem, soap and water to get rid of them. I don't think it's them but I'm not sure.

The other thing that might be the problem is that the soil is 70% peat moss so it stays wet for a while. I watered it when it was dryish 2 inches down but soon after the flowers fell off and the leaves began to yellow and fall off. Not all though. Only the smaller leaves are falling off.
Does anyone have an idea of what might be wrong with my tree?

Could you provide picture?
Edit: the new leaves curling are the leaf miners

Rand, welcome to the Citrus Growers Forum. Thank you for joining, we are glad to have you as a member. I don't think Rand542 tree is bothered with leaf miners, as the tree is in Massachusetts. The tree's problem is the medium that he has it planted in. A 70 percent peat moss based medium is a real killer. What happens when a medium is mostly peat moss, is that it compresses with every watering. The compression reduces the medium's porosity thus eliminating the root zone oxygen. At the same time the carbon dioxide that is given off by the roots, cannot escape, therefore builds up in the root zone to toxic levels. Further, a 70 percent peat moss medium holds way to much water, and the root zone stays water logged for an extended periods of time, causing root rot. A containerized citrus tree needs to be planted in a medium that has a high level of aeration, and rapid draining. A very popular medium with container growers is 4 parts CHC (Coconut Husk Chips) and 1 part peat moss. Another good medium is 4 parts pine bark chips, 1 part peat moss, 1 part coarse sand. Lastly, citrus do not have male and female flowers. Citrus flowers are complete, meaning that all citrus flowers contain both the male and female components. You need to transplant your tree into a proper medium, or the tree will die. - Millet (1,116-)

How long do most pine bark chips last in pots? CHC are expensive so i'll probably get some PBC's.
Thanks for the help. I thought the soil was the problem but I didn't know how to fix it.

I use pine bark chunks and I find it last for several years, but my bark does not contain any cambium--I use bark from dead trees.

As for the flowers, there are no male and female flowers, but some flowers on a tree can be incomplete--meaning that they typically do not have the female part (stigma) and appear to be male.

There are some citrus species with male and complete flowers. Only female flowers doesn't exist. I have a Buddha's Hand tree that does that and I know more cases.
http://www.usanpn.org/?q=Citrus_spp#info

Covrig, I believe they are not actually male flowers, but defective flowers where the flower did not successfully produce all of the required parts. Rand, some members on this forum, have reported that smaller amounts of CHC can be found at pet stores such as Pet Smart. Also many nurseries that carry orchid supplies also sell CHC. Or you can purchase them in larger amounts from The Crystal Company of Saint Louis. The main advantage of CHC over bark is that they last 4 to 5 years, and naturally have the perfect pH of 6.5. - Millet (1,115-)

That's good to know about the citrus flowers. I bought some CHC's from an orchid site online. Thanks for the advice and for saving my tree. If I need more help I'll post.

Well,
I experience mostly male flowers in fall, when trees were moved indoors and get a flower because coming from cold outside to warm inside...
Here the light conditions seem to low to promote complete flowers, and thus I have no fruits from these flowers...
But the rich fragnance of the flowers fills the room during Christmas time, wonderfull...

BTW: Happy new year