They are tough as old boots and so rewarding. Yes, by rights, you should have dealt with them sooner but, in my experience, think you will be able to get away with it. Let them flower for ever! They will cope with being tidied up in south of England in the autumn but if you are further north, leave them alone until March before you tidy them up by pruning old stems.
Rox, they tolerate being disturbed but how about an experiment of move some and leave the others if you have space and time. Then you will be able to inform us how it went!
Sorry, am a penstemon friek! :D