It would have been if there was an actual structure to the temple, but the temple is a hollowed out cave. The "top" of the temple is a flat plateau that is packed with people. Ifrit would have had to pretty much win the battle and kill almost all the Klug's remaining forces to get there safely, and how you would have gotten there unnoticed would be difficult.

Perhaps you're on an outcropping near the top of temple, but on the side of the rock formation?