Do Evening Primroses bloom as early as mid to late spring? (The ones
I've seen, don't, in the UK, but it's the annual, not the perennial,
ones that are common here.)

I'm trying to remember a specific month, but they certainly were
blooming before the black-eyed susans and other summer plants. I'm
guessing May. Maybe June.

I'd suggest Caltha palustris (Marsh Marigold). (Which as a marsh plant
might be expected to be drought sensitive.)

Easy enough for the original poster to look at both photos and
distinguish between these two. The evening primrose has more of a
single vertical stem (or a few), whereas the Marsh Marigold looks like
more of a bush shape.

But there are a lot of yellow flowers that look a bit like a buttercup
(not just in one family, either, probably because the ancestor to most
of the dicots probably looked something like a buttercup).