I remember watching Brandon Sanderson's lectures on writing when he said to ignore language.

That's frankly silly of him. At the same time, though, building an entire fictional world is a massive enough task that I think it makes sense to focus on what is needed to make the story work. My personal philosophy is to focus on the parts of the worldbuilding that you enjoy. Language isn't a particular interest of mine, but there are a lot of people who find it fascinating, and they will appreciate stories with well-developed fictional languages - just look at how many people have learned Tolkien's Elvish, or Klingon. Conversely, if someone really wants to muck about with plate tectonics and meteorology, then I say more power to them, and I wish them the best in finding likeminded people. I myself lean most heavily into worldbuilding history, politics, economics, and engineering, but I understand that those things aren't everyone's bag.