Compost may be to blame, night time temperatures of less than 7C (5 at a squeeze) definitely and lack of feed could also be the cause, because some composts run out very fast.
They may be small, but they are probably quite hardened, ready to go when conditions change.
I had a year with difficult tomatoes and it was just down to the compost. In that case it was very acidic and had no fertilisers in it. Potting up into decent compost was the answer and they recovered well. Keep trying with your tomatoes, you have invested a lot into them already.
On the other hand, it is not too late to sow again. With everything going for them, they will be ready to plant out 6 weeks after sowing, which takes us into the middle of June. As some parts of the country still have frosts in early June, a mid June planting is not too late, but you could go for faster ripening cherry tomatoes or other earlies if you sow again now.
Good luck