Part of why cats like you is probably because you don't like them and seek them out. Most cats are wary of people directly trying to approach them without invitation or greeting. In general cats will find the person ignoring them the least threatening and most approachable.

It never fails that cats tend to flock toward the person that is allergic to them, because that person is typically entirely disinterested in them lol. Not being interested in the cats is likely why you make friends with them so quickly!