I say yes!!
I have a particular set of people that actually follow me from community to community, just to downvote what I say and the articles I post. All because I posted a neutral article about the Green Party to the c/politics sub. That made them mad enough to be obsessed about me. lol
So if you are determined enough to do that, then ya shouldn't be upset that people can see what you're doing.