Remove as much of the pea gravel as possible. Stones and gravel
in garden soil are a waste of valuable space, assuming that they
are not part of a well designed drainage system which is well below
the cultivation zone of the garden. Ideally, the inorganic material
in your garden should all be in the 0.1 to 2.0 mm range. Pea gravel
(a loosely defined term!) is in the range of 4 to 18 mm (1/8" to 3/4").
Every piece that you remove improves your garden quality.

I need many tons of pea gravel for some projects and I am certain that
there is somebody in your area who is like me: Willing to come out with
a truck, some helpers, wheel barrels, screens and shovels. I'd gladly
screen and haul away that much free pea gravel. Every summer. Forever.

You should find somebody who wants the free gravel. Then stand back
and watch the other guy and his crew remove the problem gravel for free.
If you are nice, offer them some of the lemonade that you are sipping.

After you gets rid of that 3" layer of pea gravel, you should import 3" or
more of good organic matter and possibly some gypsum to augment
the existing soil. I know that you said that you have "good soil", but
it is extremely rare to find any soil that isn't worthy of improvement.
Is there such as thing as soil which is "too good"?

Personally, I believe that one will almost always reap the benefit
of any soil improvement in the very first year of gardening. If it were
my garden plot, I'd add about 38 cf of Sphagnum, 100 cf (or more)
of compost, and some gypsum if the soil has many fines (clay).

If money is an issue, then just add compost. Municipal compost is
very adequate. Yes, I'll hear opposing views on that statement.

If money is a very big issue, then add a smaller quantity of compost
and start producing your own today.

Of course, I'm a fanatic at times. I'd pull some random 1 foot deep plugs
of soil and perform my own Emerson Test on each strata of existing soil
to determine exactly what the soil composition is at each strata. And
pH testing is equally important and easy to perform. Why guess?

Remember: Gardening is fun with good soil. Gardening is work with
average soil. Gardening is a pain in the ass with poor soil.

Good luck,
Gideon