If higher-ups complain about intempestive code refactoring, it's always a good idea to stop for a moment and to start becoming less trigger-happy with refactors. It's OK to take some time to determine what actual value refactors bring to the project in tangible terms - intuition is not enough. Convincing a critical manager is a good start, because their tolerance for programmer bullshit is low if they don't actually write code.

Very often, and this is especially prevalent among junior programmers who care about what they do, the reasoning for refactoring turns out to be something along the lines of "I don't like this" or "I read some cool blog article saying things should be done that way", without any care about whether or not the change in question is actually improving anything, or, if it does, if the improvement is worth the degradation in terms of quality (new bugs)/maintainability (added genericity making the code more difficult to understand, cryptic features of the language being used that make it hard to understand what's going on, I'm sure there's other examples...)