I am no expert on matters of crop rotation and veggie diseases, but I would say, so long as you removed all the spuds, the last thing you would want are volunteers sprouting in the middle of your asparagus bed, and you enriched the ground and improved drainage, if drainage were an issue, then it should be fine!