You should start by getting the kernel compiled, because ultimately the modules need to be built with the exact same GCC version as the kernel was built with, and within the same build environment. So you'll need that for sure.
After that, essentially you just configure it to build it as a module if it's not already a module, and you can run make only on the module to speed up builds.
You may also need to build and flash your own kernel if the default one is CONFIG_MODULES=N, because if the kernel doesn't support modules, then you have no way of unloading or reloading your driver in the first place.