While Sweeper may be rather active in her mission, she's actually not that well known. Her followers my be in the hundreds (at most), and they're too spread across the land with little way to work together. The Scholars and others don't want attention to themselves, and thus keep quiet publicly, though they do speak of the Lady to friends and connections. Those who do attempt to spread the news of Sliver Sweeper tend to be ignored as a bunch of weridos - it doesn't help that their mission seems odd and few are willing to listen to some shaven-headed "fanatics".