Generally mostly by cyclomatic complexity:

• How big are the methods overall

• Do methods have a somewhat single responsibility

• How is the structure, is everything inner-connected and calling each other, or are there some levels of orchestration?

• Do they have any basic unittests, so that if I want to add anything, I can copypaste some test with an entrypoint close to my modifation to see how things are going

• Bonus: they actually have linter configuration in their project, and consistent commonly used style guidelines

If the code-structure itself is good, but the formatting is bad, I can generally just run the code though a linter that fixes all the formatting. That makes it easier to use, but probably not something I'd actually contribute PRs to

How do you learn to spot these situations before diving down the rabbit hole? Or, to put it another way, what advice would you give yourself at this stage of the learning curve?

Probably some kind of metric of "If I open this code in an IDE, and add my modification, how long will it take before I can find a suitable entrypoint, and how long before I can test my changes" - if it's like half a day of debugging and diagnostics before I even can get started trying to change anything, it's seems a bit tedious

Edit: Though also, how much time is this going to save you if you do implement it? If it saves you weeks of work once you have this feature, but it takes a couple of days, I suppose it's worth going though some tedious stuff.

But then again: I'd also check: are there other similar libraries with "higher scoring" "changeability metrics"

So in your specific case:

I wanted to modify Merlin 3d printer firmware

Is there any test with a mocked 3d printer to test this, or is this a case of compiling a custom framework, installing it on your actual printer, potentially bricking it if the framework is broken - etc etc