I need some help with my meyer lemon..

I live in zone 5 in the US, so my lemon tree is a potted plant. It
normally lives outdoors after spring frosts, and comes into the house
before fall frosts. The tree is 3-4 years old.

I got one lemon off it the second year, and last year I picked off all
3 fruit buds (silly me) checking to see how secure they were. Now the
tree is beginning to bloom again, and I am pollinating, and seeing
fruit buds again. However, the tree has dropped a bud, and this time I
wasn't playing with it, and I would like to know what's going on and
how I can get it to set fruit again.

Details..
Tree lives in my living room where it gets decent morning and afternoon
sun via south-facing picture window. Temps are coolish, maybe 60 at
night, but warm during sunny days, otherwise low-mid 60s. I am
watering when the soil is dry, generally once per week, with water.

I do have a water softener, but a young orange tree (3 years but tiny)
and a mandevilla spending the winter inside (and a few various
houseplants) show no signs of problems with the water, they all get
similiar treatment, although a tomato plant just producing did keel
right over and die on me.

Once a month I fertilize with miracle-gro, maybe a bit on the strong
side (1 tsp/liter bottle). I did put a 1/8-1/4 inch by 2 inch
houseplant fertilizer stake in the soil about 4 inches away from the
trunk when I noticed the first blossom.. not sure but I think this is
another miracle-gro product, comes in a package of 6-8; only one
inserted.

Leaves look ok, they don't seem to be dropping or turning yellow.
There are.. eleven? blossoms on the tree; one fruit bud (used to be 2),
several of the blossoms are just starting so they are tiny, maybe 3 are
mature blossoms (open). Should I pick off remaining blossoms..?

The tree is in a standard 8 inch terra-cotta clay pot, and stands 32
inches tall. Soil was recently added to the pot to cover some roots
that were exposed last time I watered (Ball brand potting soil, light
fluffy - great stuff).

What I'm calling a fruit bud is a mini-miniature green swelling that
looks to me like a pint-sized green lemon, smaller than a pea, so I'm
assuming the hand pollination worked. It forms at the base of the
blossom. The first lemon it set was pollinated during the summer, and
the fruit was picked that December. Since then, two blossom cycles
have occurred.. during the winter!

Any ideas what might be happening to the fruit, or how I can ensure
another crop..? Thanks..