Even if crime wasn't a problem, we should be allowed to protect our culture. Not every country needs to be like USA.

Culture always changes. The culture of your country has not been like this since the dawn of time. There is no good reason, why it should stay just the way it is right now, only because "that's the way it always has been" in your memory. Also if the newly arriving people make all of your felloelw countrypeople abandon their old ways, maybe their was something wrong with those traditions to begin with. If you are only worried because the new people will bring their own culture and stick to it, that just adds to the culture and doesn't take yours away.

And I'm not even sure why I have to defend myself.

I personally think one needs really good reasons if one chooses to defend xenophobic policies and puts millions of people under the general suspicion of spreading crime and violence while nearly all of them are just trying to get away from the war and violence in the countries where they are coming from.