If it's the pollen he is allergic too (and I guess it usually is with hayfever) it will be the wind pollinated flowers that will set him off. Supersprout is right that patch tests would confirm an allergy but I think it unlikely a GP would arrange that unless the hayfever were so severe that it disabled him, in which case he'd probably have had them by now. If the pollen is visible in the flower and blowing off in a breeze it might be a suspect. FAR more likely to be grasses than your sweet peas!!
Love and compost
Linda (highly allergic to Willow, Alder, Birch and grass pollen, OK with every other flower)