Many years ago, when I was digging my first vegetable garden, I rented a
rototiller to remove the lawn and ended up regretting it. The machine
inverted clumps of grass which still had to be overturned by hand in order
shake off the useful soil. If (in a fantasy world), it had NEATLY overturned
them exactly 180 degrees so the grass simply died, that would've been great,
but it was a fairly random mess. I ended up finishing the thing by hand,
which wasn't TOO awful, since I'd just bought a really nice spade & fork.

At my new house, I'm about to create another vegetable garden. Again, I have
excellent tools, but I'm wondering about renting a tiller, since I'm dealing
with a 20x40 foot area. However, I'd like to avoid the same mess. Was the
previous fiasco due to the type of tiller, the way I used it, the position
of the moon, or what? Combination of all factors?

All tillers mix the layers of soil and whatever cover is on it. The only thing
that that will turn the soil over 180 degrees is a spade/spading fork or a
large turning plow (16 or 18 inch bottoms)
Tilling is not usually a major problem if you go over the ground several times
to thoroughly incorporate the plant debris.