For most people, except sub Saharan Africans, we are also talking about our ancestors when talking about neanderthals. Most of those bones we see on museums are probably the great x grandfathers of many people walking past.
Obviously we have no idea what happened over huge parts of deep human pasts, Neanderthals were a sparse population to begin with, and absorbing their people into the rest of humanity just by fucking is certainly a solution
Alternative view: that age group is the most paranoid giving answers to pollsters.