Don't apply scale twice when using fractional scaling #5650

Job	Run time
build-and-test (unstable)	54s
fedora	1m 29s
build-and-test (stable)	1m 3s
build-and-test (development-target)	56s
open-suse-leap	3m 13s
lint	11s
	7m 46s

Provide feedback