The device can be unbound from the driver via sysfs, so regardless of
whether the driver is builtin or a module, its .remove() function needs
to stick around.

Signed-off-by: Thierry Reding <treding@nvidia.com>