Olives are basically a cultivated plant, grown from cuttings. I doubt they are being lifted from the wild. There may be an issue with groves being bought up for redevelopment but you could regard your purchase as a rescue then. The biggest threat is a disease (xylella) that is a very big problem on the continent.
I have a couple in deep 12" pots that are about 3' 6" tall - more like bushes than trees, I need to prune them into the right shape really.
I bought them some ten years ago when they were about 12" tall and looking rather sad on the remaindered bench at the garden centre. I couldn't leave them to go into the bin. Just a few quid the pair. They flower and produce a few small olives in hot summers though the blackbirds take them as they ripen.
Yes, something that is more finished will cost you from £50 up to really silly money!