drm/radeon: improve PLL limit handling in post div calculation

This improves the PLL parameters when we work at
the limits of the allowed ranges.

Signed-off-by: Christian König <christian.koenig@amd.com>