Most likely they're mining bees which nest in hollow stems and the like; they can adapt to narrow spaces under tiles and the like quite easily. If they're carpenter bees they'll be chewing the wood away, but it doesn't sound like it, and they only tunnel into wood which is already rotten. They only have one generation a year, so they'll build their nests and disappear.