clk: imx: drop the dependency on ARM64 for i.MX8M

Moving to support aarch32 mode on aarch64 hardware, need to drop
the dependency on ARM64 to make the driver could be selected for
ARM32.

Signed-off-by: Peng Fan <peng.fan@nxp.com>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Stephen Boyd <sboyd@kernel.org>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>