Cyber Resiliency Board Briefing 13: The Rehearsal Problem: Why Most Cyber Exercises Confirm What the Organisation Already Believes
Most cyber tabletop exercises and recovery tests are designed to be passed. They test whether systems can be restored and whether the playbook reads well, and they are run by people who already know the answers and frequently have no experience of dealing with a real destructive attack.
The question that decides whether an organisation survives a destructive attack is who may decide what, at 3am, with half the facts and the usual approvers unreachable, and very few exercises ever put that question to anyone.
The dress rehearsal with no understudy
Boards have been told, rightly, to rehearse. Most now receive an annual assurance that a tabletop was held, a recovery test succeeded and lessons were captured. Regulators expect it under DORA and NIS2, insurers ask for it at renewal, and the audit committee records it.
Think of a theatre's dress rehearsal. The cast knows the lines, the set is built, and when someone stumbles the director calls "go again". It is a sound way to get a show ready. It tells you nothing about the night the lead falls ill at the interval and the understudy has never been on stage.
Most cyber exercises are dress rehearsals, and four habits keep them that way.
The team being tested chooses the scenario. Nobody sets out to embarrass themselves, so the scenario lands somewhere the team already feels competent.
The injects follow the plan. Identity services still work, email still works, the chief executive answers the phone and the backup catalogue is intact. The exercise quietly inherits every assumption the plan was written on, which means it cannot find the ones that are wrong.
Success is measured as restoration. "Tier-one application restored in six hours" is a useful engineering result. It was achieved in a clean environment, with no adversary still inside, and without anyone having to decide whether it was safe to reconnect.
The facilitator has never been in the room for real. Many exercises are run by capable facilitators who have never lived through a destructive attack. They can keep the script moving, but they cannot supply the confusion, fatigue and contradictory information of hour nine, so the exercise never feels like the real thing.
The outcome is an exercise that confirms what the organisation already believes. Confidence rises while exposure stays exactly where it was. I have watched an organisation with flawless recovery tests spend the first eleven hours of a real incident establishing who could authorise taking its payments platform offline. The technology was ready. The authority to use it was not.
Rehearse the decision as well as the restore
Technical recovery testing should carry on as an engineering discipline. The exercise the board sponsors has a different job: to prove that the people who will actually be in the room can make consequential decisions, within agreed limits, without waiting for someone who is not there. That is delegated authority, and it only reveals itself under pressure. Four design choices get you there.
Remove people. Declare the chief executive, the CFO, the Active Directory administrator and the named incident lead unavailable before the first inject. Watch whether their deputies know they hold authority, and whether everyone else accepts it.
Remove the tools the plan depends on. No corporate email, no collaboration platform, no single sign-on, no plan stored on the file share. If an out-of-band channel exists, make the team use it for real.
Force trade-offs with no right answer. Isolate a revenue-critical service on incomplete evidence. Notify a regulator before the facts are confirmed. Reconnect a restored system that has not yet been verified clean. Each inject should cross a threshold where a named person must own the risk. Anything touching extortion or disclosure should involve counsel in the design, since the legal position varies by jurisdiction.
Measure decisions, not restore times. Record who decided, on what evidence, under which delegation, how long it took and whether it was written down. The metric that matters is the time from "a decision is needed" to "a decision is made and owned".
Two conditions make the design honest. The scenario should be written by someone independent of the team being tested, much as a red team is independent of the defenders. And the exercise must be allowed to fail. An exercise nobody can fail is a product demonstration.
Questions for the board
When did we last exercise with our principal decision-makers deliberately removed?
For our five most consequential crisis decisions, who holds the authority, what are its limits, and who is the named deputy?
Who designed the last scenario? Were they independent of the team being tested, have they are real-World experience of dealing with a destructive cyberattack?
Did the last exercise force a decision this board would find uncomfortable? If not, why not?
What changed in our delegations of authority as a result?
The final question is the real test. An exercise that changes nothing in how authority is written, held and delegated has rehearsed the organisation's comfort, and comfort is not a recovery capability.
The purpose of a rehearsal is to find the failure while it is still cheap. When an exercise finds nothing, the only thing it has told you about is the exercise.
