As the home-manager depends on the actual used nixpkgs version and is not always used, remove it from the base dependencies but use still use it implictely if a home-manager input exists.