There are a couple of reasons that might not work:

• Maybe we'll asymptotically approach a point that is lower than human-level cognitive capabilities
• Gradual improvements are susceptible to getting stuck in a local maxima. This is a problem in evolution as well. A lot of animals could in theory evolve, say, human level intelligence in principle, but to reach that point they'd have to go through a bunch of intermediate steps that lead to worse fitness. Gradual scientific improvements are a bit like evolution in this way.
• We also lose knowledge over time. Something as dramatic as a nuclear war would significantly set back the progress in developing AGI, but something less dramatic might also lead to us forgetting things that we've already learned.

To be clear, most of the arguments I'm making aren't really about AGI specifically but about humanities capability to develop arbitrary in principle feasible technologies in general.