I'm a long distance road biker, I'd say upgrading the saddle and getting some reasonably comfy padded tights is well worth the money. You don't have to be in spandex.
Saddle currently: Selle TRK (pick a width based on how wide your sitz bones are) Bibs: you gotta decide here. I might recommend Patagonia's MTB stuff if you want something fairly multipurpose that works with camping (if you've got the money)