Unfortunately the media did a character assassination job on rainwear, depicting its wearers as silly, or weak, or perverted, and there's still many people who think of it in those terms.
I have a mate who can think of sex only as filth, dirt, smut, sleaze or pornography and smears plastic and rubber as having the same meaning as all those.