Just to make sure you can do a worm audit:
Outline and remove a block of soil about a foot square and 7 inches deep
Spread the soil on a board and as you break up the clumps of soil, gently extract and count the worms. Little ones score the same as big ones!
You would hope for around 60 worms.
If you have fewer, just keep adding organic matter to the soil and the ones that are there will happily make new worms :)