Well, thus far everyone in the Librarian faction seems to either just owe loyalty to the Librarian, or is motivated to bring down the system by the death of loved ones. Including Angelo. So yeah, not scared of them, since the 'revenge' motive will burn out pretty fast under pressure.