Note there's a group of users that larger than the group of users without JS (for whatever reason): users of assistive technology. And they don't even have a choice.
While I'm all for considering the needs of every user... If you get to the point where you're worrying about no-JS users, I hope you've already considered the needs of people with disabilities, whether temporary or permanent.
Edit: oh right, wanted to add: just making a site work without JS doesn't automatically make it accessible to people with special needs.