In what reality is it important to pull someone's card. You can ask anybody wearing a shirt what their favorite song is or if they've seen them live. That's actually acceptable, because you're interested in their subjective experiences. Publically announcing to someone that you want to out-group them over a trivial matter is not.

Edit: and generally, that's how men interact with each other. But some take the existence of a woman in their fandom to be a direct attack on their already incredibly fragile masculinity.