I would not worry about what the tubers look like at the moment, you do not harvest them until a lot later when the plant had died back and new tubers have formed.
When the plants have died back, cut the stems down to 1-2ft so you know where they are, and then just dig up any you want during the autumn/winter.