I started reading about Shintoism after this manga, so I'm not sure if I interpreted it correctly (or maybe there's no "correct" interpretation).
.
In Shintoism, a shrine is where the god resides and it stores a sacred object representing the god. While the shrine is usually a building, it could pretty much be anything (a rock, a tree, a mountain), as long as it's important to the god.
.
In this case, the sacred object is the gemstone. And what happened is that the MC became the shrine of the god by keeping the stone, so his friend is now residing in him (or with him). The friend became his personal god instead of a god to bless everyone and be worshipped by everyone.
I don't know how long it has been since I first read this, but my old comment was:
Exclusive ass, public mouth
And I'm laughing so hard because it was a very accurate summary