Hi
Problem could a mixture of things.... Adult Vine Weevil (the holes/notches in the leaves) though as stated below, it's the weevil larvae that are the worst pest, they loooove potted plants! The tiny flies might be a red herring (sorry!) could be sciarid/fungus gnat.... more a problem for seedlings than 'fully grown' plants. I've had a huge problem with whitefly this year both inside and out!
Anyhow, suggest you visit one of the garden pest control sites - for example www.buzzorganics.co.uk - for more information. These days there are lots and lots of them around with many potential solutions for you to use (beneficial bugs or 'green'/environmentally safe solutions)!
Keep smiling and more to the point don't give up you can win!