Racism is an extreme form of Classism which is caste-based.
Social class distinctions can vary by time and place and has been an issue among humans at least since the advent of agriculture. Social class may be determined by wealth, pedigree, politics, appearance, language, and likely hundreds of other variables depending on the social constructs of the given society.
Racism is derived from 17th-century European pseudoscience which proposed that there are "races" of humans which determine their intelligence, values, and appearance. According to the philosophy of racism which has been completely debunked as science for over a century, "White" people, the dominant caste, do not have inherent racialized characteristics while all other racialized groups (which are different over place and time) are born with certain characteristics that may or may not be tolerated depending on how liberal the racist is. I can't describe here the rules of why a person may be labeled a given race because it's as complicated as the entire field of sociology and unique to time and place.
Personally, I don't think any kind of classism is legitimate. That being said the majority of people throughout the world believe that at least some forms of classism are legitimate so it's not a question of allowing it, rather choosing how to deal with people who may discriminate against you for arbitrary and typically non-sensical reasons.