I grew acorns last year, and have them again this, and they are going great guns. As periwinkle suggested, I think the problem is poor pollination. Last year I had masses that were big and firm (in fact I still have a yellow acorn in my conservatory!) plus, towards the end of the season, I started picking the smaller ones whilst still tender and roasting them whole!