I am very new to this herb growing having just planted a window box
(outside) with oregano, parsley & thyme.

The oregano & thyme seem fine but the parsly seems to have attracted
some small black insects which are living on the base of the stems!
Are these anything to be concerned about and if so what is the best
way to get rid (preferably without a pesticide)?

cheers

If they're aphids, squish them with your fingers then sprinkle with
water to wash away the dead bodies. Do it quick before anyone sees...