If you are in London, the clay there is quite good for growing fruit anyway, the rubble shouldn't hurt too apples too much. There may be a problem in the situation you state for the soft fruit - the raspberries tend to like it moister and with plenty of muck, the redcurrants and whitecurrants tend to like a lighter soil thats fertile but well drained, so it may be hard to achieve all that in the same area if it's a small area.