If I had to guess, I'd say that they were talking about how it's barely affordable just to live in some places (wage slaves), and moving yourself and your family to an entire other state takes substantial time and money. Not to mention the emotional difficulty of leaving everyone and everything you know.

The same argument was made about recently freed slaves who had nowhere to go afterward, and how they should just move if they don't like the continuing slavery-like treatment where they lived.

It does seem very cold and privileged from that perspective to tell someone to just up and leave, but this specific comparison seems unnecessarily impassioned