I use Zimbra with an external email gateway that only accepts authenticated email. Zimbra is pretty heavy (it's intended to be a Microsoft Exchange replacement) but it at least has a huge amount of protection built-in to deal with spam and comes configured out of the box to not relay (well, outside of you setting up aliases and lists.)

That said, it's not hard to find "incoming email only" configurations that deliver to local mailboxes only, for most email servers. The thing to avoid is having a single server configuration that tries to do both - accepting external email and sending locally originated email out. The configurations do exist to do that, but they're confusing and tricky.

External email gateways... that bit is hard. I use a mail server I set up myself on a VPS. It does not listen on incoming port 25. It requires credentials. I did this largely because I was trying to send email out via Xfinity's customer email relay, but the latter kept upping the authentication requirements until one day Zimbra just couldn't be configured to use it any more. And each time they changed something, I wouldn't find out until I noticed people had clearly not received the emails I've sent out.

VPSes are problematic as some IPs are blocked due to spam. There's not much you can do about it if you're stuck with a bad IP, so if you can find a way to send outgoing email via your ISP's outgoing email server, do that. For Postfix, you can send out authenticated email using something like: in main.cf:

relayhost = [smtp.office365.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes

and in /etc/postfix/sasl_passwd:

[smtp.office365.com]:587 example@outlook.com:hunter2

So in summary:

• Consider an email-in-a-box solution like Zimbra, I understand the wish to go for something light but it might make sense if your aim is just to control your own email
• Regardless of whether you do or not, use separate servers for incoming/outgoing email.
• For incoming email, lock it down to accept local email down if you're manually doing this rather than using an email-in-a-box solution like Zimbra.
• For outgoing email, use authentication and avoid it listening on port 25. Consider either directly using your ISPs, or if that's not practical, configuring your outgoing email server to relay in turn to your ISP (see above for how to do this.)

Good luck.