Star Trek and X-Men were a huge part of my childhood and teen years as well, so I 100% get where you're coming from.
Only tangentially related, I really do wish Marvel would take a back to basics approach with the X-Men so I could get into them again. I know a lot of people love the Krokoa era, and I'm happy they have it, but for me it is too dense and too far removed from what I grew up with for me to be able to penetrate. I know they've got an upcoming sea change happening, so maybe then.
From the opening log of "Whom Gods Destroy":
So, at least this one TOS episode indicates that there is only one small facility which the Federation uses to house all the remaining criminally insane people in the galaxy. I think we can assume that by the galaxy, Kirk actually means the Federation. But as of that era, there apparently exists a medication that they believe will cure people of mental illness.
How much stock we want to put in one third season TOS episode I think can be debated -- and crucially we never get any confirmation as to the long term success of the medication -- but it is part of the canon.
There is also the Tantalus V penal colony from "Dagger of the Mind". Before they beam down, Kirk tells McCoy that it's more like a resort colony than a cage, though the doctor who ran the facility was using a machine to essentially brainwash both inmates and staff.
As for incarceration and rehabilitation in the 24th century, we know Tom Paris was at the New Zealand Penal Settlement when Janeway sprung him, with the approval of the Rehab Commission. When we see the settlement, the prisoners appear to be doing some sort of labour: one is carrying something, and Paris appears to be calibrating some sort of machinery. Granted, we don't know exactly what he was doing or why. Maybe he was working on a project he volunteered for or even conceived himself, and was given access to the resources to carry it out.
Ro Laren was on the Jaros II penal colony after her court martial. She was sprung from that by Admiral Kennelly, and he claims it was difficult to do so.
Kasidy Yates was incarcerated for six months for aiding the Maquis, though there's never any indication that the sentence isn't purely punitive.
In "Blaze of Glory" we saw that after his capture in "For the Uniform", Michael Eddington was being held aboard a station in a fairly small cell. He was still wearing civilian clothes. It's possible he hadn't yet been formally tried and convicted, though.