No one has to know
You’re right. No one has to know; however, spouses (and even kids) have an uncanny knack at figuring these things out. But even if you manage to pull the wool over other people’s eyes, you’ll know. And you ruminating over it (positively or negatively) can constantly draw your attention away from where it needs to be—on your spouse, children, house, job, church, etc. Allowing a fantasy to become a reality is like renting out your brain to the person involved. And like renters from hell, sometimes they refuse to leave when you’re ready to have your brain back.