Servants have ruled; rather, slaves. The Babylonians in general might be called slaves, by comparison with the "kingdom of priests" (Exodus 19:6), and the "sons" of Jehovah (Isaiah 45:11; Hosea 1:10).
Or the expression may mean that even baseborn hangers on of the conquering host assumed the right to command the defenceless captives.