There's the fact that the Sweepers are now allied up with the Black Guild. So it's best to not go around making enemies.