Okay this actually touches on something interesting, but before I get to that, I think it's perhaps a sign that your intuition with English is very much at odds with (not merely different from, but directly contradictory to) the broader English speaking community. Most English speakers would be either 100% okay with "you" being either singular or plural, or would be more likely to interpret it as more singular. That's why some dialects have developed terms like "y'all" or "yous", and why phrases like "you all" or "you guys" get used.

But the really interesting thing here is that etymologically, you are much closer to the mark. Historically English has had a number of different second person pronouns. You, ye, thou, thee. "You" was, in fact, the plural objective second person pronoun. Today, English never distinguishes between subjective and objective pronouns in the second person, even in common informal use. But it used to have "thou" and "ye" as the singular and plural (respectively) subjective pronouns, and "thee" and "you" as the objective. And I find it kinda funny how that's basically flipped in modern usage.