Unit size had less to do with it, people were already operating massive ships and soldiers. I believe the concerns raised were more to do with having over 89 galaxies under their belt, and there were a lot of scares over power.

And yet, shortly after Rogue left they were not over the top at all. By the end we had millions of ships in every battle and insane sun exploding magic and black hole guns... Yeah.