The root of the problem is that it's an indirect reference to an individual. They/them is commonly (until very recently) referring to a party (singular or plural) that isn't present. When you use it as a direct reference to someone who is present, most people feel like it's incorrect because of the common usage of the term being indirect.

When speaking to someone about Joe: "Joe doesn't know what they're talking about" While directly: "Joe, you don't know what you're talking about"

Both are correct, and possibly the most correct forms of the statements. Substitute Joe for whatever name and it still works. Meanwhile, it's uncommon, in Joe's presence, when not taking to Joe, to refer to (assuming Joe is using gendered pronouns) him as a he/him. "Joe doesn't know what he's talking about"

Both cases are singular, but the difference of Joe being there changes "they" to "he", and not taking directly to Joe changes "you" to "he".

The problem isn't plural vs singular, the problem is direct vs indirect reference.