I have the same problem growing flowers in trays or pots, and I prefer to sow straight in the ground. I think my favourite flower must be single-flowered hollyhocks, and I've always sowed the seed straight into the ground - and I mean sprinkling seeds about a bit, I don't do prepared nursery beds either!
At my old allotment they grew easily, and were no trouble at all. But I brought seed with me when I moved to Hastings, and they haven't been as successful.
They seemed very happy in the dry, poor soil of the old allotment. They can tolerate quite shallow or small amounts of soil too. Perhaps they don't like clay? Or moist soil? What type of soil have you been trying to grow them in?