The only fool proof way to know is have a furtle/grope around the soil to see the size of the tubers. I start to look at around 80 days for first earlies.
Removing the flowers is recommended by some, mainly for main crops not earlies, and I have tried it but not seen any noticeable difference in yield.
Jerry