The tax loophole was a perceived reason, but it was never proven to be true. The closest admission to it was Sony losing a lawsuit related to the situation and Sony admitting that something related to Yabasic was intended to skirt EU tax law, not Linux for PS2. I never heard of Yabasic before today.
Fun fact, Geohot owns a company that jailbreaks cars and their adaptive cruise control systems now.
Nope, this was an official, legally sold product. Came with a keyboard, mouse, 40gb HDD, and PS2 ethernet adapter. Required an 8MB memory card to function and installed via command line. It's a pretty cool piece of history.