Much of this is dependant on the magicians in your realm. In D&D for example, there is no spell that actually will do the job. Even "rock to mud" holds the cause that it only is effective against "unworked stone".
For the most part, advanced intelligence is the best defense against magic. Scry your enemies as best as possible, cast runes or bones to foretell the future. Whatever it takes. Also ensure nobody walks off with a bit of your castle wall. Your enemies could build a miniature version of the castle using that small bit of the wall you stole and you have a sympathetic link for magical rituals allowing you to cast spells against the wall from hundreds of miles away!
In many cases, those without magic of their own, may still use magic through evocation in the form of ritual, sacrifice, or crafting. By introducing proper runes or symbols into the stonework mere mortals could evoke the force of a god, demon, or something more elemental to empower the bulwarks to resist such attacks.
Additionally, one might have magic resistant materials occurring naturally in the world. Maybe its certain stones, maybe it involves a special metallic ornamentation or perhaps just infusing the mortar between the stones with certain herbs might do the trick.
You could also bind spirits, like a Djinn or a greater demon and force it to patrol the castle astrally and ensure nobody tries anything nasty without it being counterspelled and the local magicians being informed.