Maintaining one's masculinity is very important for almost all guys. It can override many other instincts and wishes, including self-preservation.
In humanity, men are strong, and women are weak. When a woman believes in spirituality, nobody thinks anything of it. It is in women's nature to be naive and to believe in fantasy. They are not expected to do anything, but simply to have things done for them by men.
However, when you say that spirituality is unscientific and imaginary, you're implying that anyone who believes in it is less masculine. That, if he had a problem in the material/real world, such, say, his wife or child being attacked, he would be less able to live up to his role as a man, and defend them because he is weak and lives in an imaginary world, like a woman. Only women are allowed to believe in things beyond their control. Men are supposed to be cynical, realistic, omnipotent.
What is the average young man supposed to think when everyone tells him he is weak, stupid, childish, unmasculine, for believing in a sky daddy and "fairy tales"? In order to preserve all his pride and reputation, he will of course renounce spirituality, consciously or subconsciously.