Heck, sometimes someone comes to me and asks if some system can solve something they just thought of. Sometimes, albeit very rarely, it just works perfectly, no code changes required.

Not going to argue that my code is artificial intelligence, but huge AI models obviously has a higher odds of getting something random correct, just because it correlates.

You can’t actually teach it anything new, or to put it better it can’t actually derive conclusions by itself and improve in such way

That is true, at least after training. They don't have any long-term memory. Short term you can teach them simple games, though.

Of course, this always goes into Chinese room territory. Is simply replicating intelligent behavior not enough to be equivalent to it? I like to remind people we're just a chemical reaction ourselves, according to all our science.