The short answer is yes, you may prune Mahonia, quite severely too, if you need to, but that does not really answer the question as to why it is growing so poorly. Ours grow a good 2 to 3 feet every year. They do like a bit of shade, dappled for preference as they are woodland shrubs, perhaps yours is getting too much direct sunlight?