Sorry if I sound like the sky is falling, but if you have the variety
of yellow jackets that build nests in the ground, I would try to find
them and kill them at night while they are on the nest.

Accidentally disturbing one of these nests can cause lots of stings in
a very short time, especially if the person/child runs.

Regards,

Hal

Sounds like a nest to me as well, which as you say, can be quite
dangerous if you have reactions. You do need to find out if they have
a nest nearby. Following them is the easiest way. They really aren't
that aggressive away from the nest. If you get stung away form the nest
it is usually because you have trapped them against yourself while they
are trying to share your pick nick lunch.
While I don't normally recommend killing out of hand, the best way to
get rid of ground dwelling yellow jackets is to place a large clear
jar/bowl over the opening. It must be clear otherwise they just tunnel
around. You must get it snug to the ground or they go under. They
aren't smart enough to figure out that they need to dig out, because the
bowl is clear. They will starve quickly.
Putting the bowl over the nest can be tricky; do it at night when they
are less active.
They like nice loose soil, like in your garden, but will tunnel anywhere.
Mark B.