A person will be called to account on Judgment Day for every permissible thing he might have enjoyed but did not.
From "How Can It Be All Right When Everything Is All Wrong?" by Lewis B. Smedes:
What is joy? I suspect that, to answer this question, each of us has to locate a moment in his or her life, a moment of special joy, seize hold of it, examine it, and let it be a parable of joy. My own joy parable came one night after hearing a concert by Isaac Stern and the Los Angeles Philharmonic. He played one of the romantic concertos that set my heart on fire, and I was deeply moved. So were we all. We heard him. We received his gift, and when he was finished, we blessed him. We gave him the hallowed benedictions of our sweet applause. He took joy in our blessing, and he kept coming back for more. And when he came back, we took to our feet. We were swept away, beside ourselves with gratitude, and suddenly I realized that I was enjoying the applause more than I had enjoyed the concert. And I knew why.
In the receiving of Stern's gift, and in the giving of our blessing, we were enacting a parable of the meaning of life under God in this universe. We had received a truly great gift - the gift of Brahms's genius transformed by Stern's artistry. In turn, we were moved by the feelings of gratitude for the gift and a desire to bless the giver. Heaven might be fun after all - even those endless doxologies in the heavenly choir!
Joy is an intermezzo of gratitude that interrupts the routine motion of life. Our lives, for the most part, are motion and struggle. Most of us spend our time crawling, groping, climbing, sometimes running, but always moving like the works of a clock. But now and then joy comes to arrest the motion; it stops the tedious ticking of our life-clocks with the bracing discovery that we have received a gift. It works most magnificently when we feel our own life as if it were God's gift to us. Stilling, for a moment anyway, the haunting anxiety that maybe life is made only of the stuff that hurts and angers and makes us feel small and stupid and phony. There comes a sense that life-now, here, today-is a gift worth blessing God for. When it comes, when this sense of being a gift comes, joy has come to us.
Joy is not just the experience of God, thank God, though being with him, in the sight of his beauty, will be the ultimate joy. . . . There is an earthly joy, a joy of the outer as well as the inner self, the joy of dancing as well as kneeling, the joy of playing as well as praying. Any moment that opens us up to the reality that life is good is a parable of the supreme end for which we were made.