Remove the obsolete fb unreferencing system that is no longer used
since we've transitioned to atomic modeset.

Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>