Hi, arborist here! Fallen leaves are generally pretty low in nutrients--trees send back as much as they can into the tree before dropping them. So, you probably don't need to fertilize shade trees whether they stay or go. The bigger issue is that removing a huge input of carbon into the soil leaves you with low organic matter and lowered ability to retain what nutrients do exist. This poor soil quality will stress your trees, but adding fertilizer doesn't address the root issue.
Fruit trees generally do need to be fertilized since fruits are much higher in nutrients and are not generally returned to the soil unless you're doing some kind of poo composting scheme.