Like you said, your plan involves building a four-family home. That's labor and worth fair remuneration. It's just that, in order to get that remuneration you'd be taking payment from tenants who build no equity for their money. Yeah, you'll have to renovate in 30 years, but you'd still have property and the money paid in rent while they don't.

A landlord can also simultaneously do valuable work supervising and managing a property. That's not mutually exclusive with profiting from ownership, and we can separate how we evaluate the two. It even comes up with billionaires: Bill Gates obviously did work worth payment as CEO of Microsoft, it's just not where he got most of his fortune. It can simultaneously be true that he's a talented guy who deserved to be paid, but most of his fortune came from exploitative business practices and profiting off of the labor of others.

Also, to be clear, there's a difference between structural and individual criticism. Obviously slumlords are pieces of shit, but there's a difference between that and someone who really does work as a property manager doing right by their tenants, or a family renting out a part of their home to make ends meet. I can think that landlords should be judged on an individual basis, while landlording as a thing shouldn't exist.