Yes, same here. Straggliness is the main problem.I don't bother with the growing new plants business, just leave them from year to year. I don't try to let them seed, so I chop the flower stems back quite hard after flowering.
Sometimes the main stems get quite woody, but you can cut them right back to new growth around the base. And sometimes the woodier plants just pack up altogether.

If you're growing just one or two wallflowers in a mixed border (which I do, as an early scent provider on the sunny terrace just outside the back door), then growing them as perennials seems perfectly viable. And you get a few stray flowers at this time of year if you're lucky.

It seems to me that a lot of what is passed on to amateur gardeners as gardening advice has originated from large scale gardening operations, either nurseries or gardens of large houses, and it's worth challenging. (For example, was it Charlie who told us recently that leaving two half leaves, not one whole one, at the top of a cutting was to make it easier to hold when you were doing several hundred cuttings?)

You can see that if you're filling a large bed with wallflowers, it's not going to be practical to go through the plants, prune back some, take out other plants altogether, and then fill the gaps. It's much more straightforward simply to replace the lot.

But if you're gardening on a tiny plot, it is feasible to deal with plants on an individual basis, and is quite rewarding.

What I like about urg is that so many urglers challenge the conventional wisdom and can then pass on their findings to the rest of us.