@TurboshitterI'm not sure if nerfing him by taking away his spells would be fair. I'm just going to have to give him little opportunity to do so otherwise this will be the Salem witch hunts featuring blood fortress Andromeda.
Huehuehue. Well, if it helps clarify things, your Archer example was spot on, but while touching Togami's butler might give you a brief idea of the layout of Togami's fortress (assuming any of the butlers are even privvy to that knowledge), it wouldn't inform you about how any of the magic worked or even what the magic was if the butler hadn't already been warned about or personally experienced any of the traps. All you'd really know is what paths the butlers take through the fortress and maybe passwords and the location of some keys/keycards if you're lucky.
It's not omnipotent, and he can't glean an object's entire history just by touching it for five seconds. The more detail he wants, the more he needs time to concentrate. A good comparison would be with Heaven's Door from JJBA, just without the mind control. Any object he touches is an open book but he doesn't have a glossary, index, table of contents or a search key, and it takes time to read unless you're just skimming for the footnotes.
Plus, y'know, blinding him renders the Pure Eyes permanently unusable, as gruesome as that may be.