We should only call init_common_infrastructure and
init_common_devices from init_early.

Signed-off-by: Tony Lindgren <tony@atomide.com>