Not just as easy. There's a lot of room for someone to say "this was actually just metaphor" or even "these are just stories to convey values".

Take the tower of Babel, for example, we know it never happened. However, a more progressive Christian or Jewish tradition can use the story to talk about how sometimes cultural differences are simply surface level, we are all ultimately the same people. Mormons aren't so lucky because the book of Mormon was pitched as a literal history and part of the book has literal refugees from the tower of Babel.

Unlike the Bible, we have the author of the religion who very well documented how literal everything is. We don't even know who authored nearly any book in the Bible or their motivations.

I'm not arguing for a god, I'm an atheist exmo. However, there's a pretty big difference between a bunch of old stories compiled together into a book and a book of fiction that the author went out of his way to claim was "the most correct book ever written".