If the leaves turn yellow and start getting black,rotting patches then yes ,remove them but while they are still green it is ok to leave them.Odd that you are still getting buds.All mine stay now on the bottom of the ponds all year round,i have discovered they do much better that way.I only remove the leaves,even in winter,if they look manky.