It is the kinda fear that speaks to a downwards mobile middle class white guy. So that is why it was the scariest thing to literature types. Alot of work in that genera specifically of modern times point put thst is just the minority experience in America. One I like recently, the winter's tide series, has the story of innsmouth folk getting sent to internment camps in ww2 along side the japanese. The author is Ruthanna Emry and it is really good if you are interested in a modern take on lovecrafts stuff.