You still have time to clear an area for the spuds, and they will do well in most soils with a bit of compost or manure around them - they grow fast and help shade out the weeds on new ground, and earthing digging them up breaks the ground.
Like KT says, you would be best to use the bed for something else - maybe a seed bed for your seedlings?