When a destructive cyberattack hits, every board asks how quickly the organisation can be back. It is a fair question with a dangerous answer, because the fastest restore can be the start of the longest recovery.
The hospital that recovered twelve times
I once dealt with a hospital that was hit by ransomware. Some of its backups were encrypted along with everything else. The rest survived, because someone had made sure the copies that mattered were immutable, held in a vault, protected by separation of administrative duties, and reachable through break-glass access the attacker didn’t control.
Most vendors would end the story there. The data was safe, so recovery should have been straightforward.
The incident response team brought in through the cyber insurance policy approached it the way most organisations approach an outage. They treated it as a disaster recovery exercise: find the last good backup, restore the systems, get the hospital running again.
So they restored. The attacker came back. They restored again. The attacker came back again.
They did this more than twelve times.
If you have seen Groundhog Day, you know the shape of it. Bill Murray wakes up to the same morning, again and again, and nothing changes until he does. The hospital’s first team never got that far. Each restore pulled the attacker’s hidden access (what the industry calls persistence) back into production along with the data, because nobody had established how the attacker got in, what they had left behind, or which vulnerabilities they had used to do it. Each failed recovery pushed the recovery time objective a little further down the road, and in a hospital that road runs through patient care.
On a status report, every one of those restores would have looked like progress. Systems restored: green. Time to restore: within target. The organisation was measuring the thing it was good at, and the thing it was good at was making the problem come back.
The outcome changed when a different team took a different approach. They stood up a clean room: an isolated environment the attacker couldn’t reach, where systems could be examined and rebuilt before anything touched production. They investigated until they understood the root cause well enough to act on it. They removed the persistence mechanisms and closed the vulnerabilities that had let the attacker in. Only then did they recover systems back into production, and the systems recovered that way stayed recovered.
The attacker gets your speed too
The first team optimised for speed, and the attacker got the same speed. Every shortcut in the recovery was a shortcut back in.
This is the uncomfortable truth behind most recovery metrics. A recovery time objective measures how quickly you can put a system back. It says nothing about whether the system you put back can be trusted. In a flood, a fire or a failed storage array, that distinction barely matters, because the water doesn’t follow your data onto the new server. A human adversary does, and the better ones deliberately leave access behind that is designed to survive exactly the kind of restore you are about to perform.
There is a second trap hiding in the phrase “last good backup”. Attackers rarely detonate on the day they arrive. They spend time inside the network first, finding the valuable systems, the administrator accounts and the backups. A copy taken the day before the encryption may already contain everything the attacker put in place. Picking a restore point is an investigative question, and it needs an answer from the people investigating.
So the fastest restore can easily be the longest recovery. Each reinfection resets the clock, and it resets more than the clock. It costs credibility with regulators who want to know you are in control, with insurers deciding what they will cover, with staff who were told the systems were safe, and with the customers or patients who were told the same thing.
The question boards inherited
Boards ask “how fast can we be back?” because business continuity taught them to ask it. The answer lives in the business impact analysis, expressed as a recovery time objective for each application, agreed years ago with a backup and disaster recovery team that was planning for a different kind of disaster.
Those numbers still matter, because they tell you what the business can tolerate. The trouble is that they have quietly become the definition of success, and the measure has started to drive the behaviour. If the only recovery number in the board pack is time to restore, the incident team will deliver time to restore, and they will be praised for it right up to the moment the attacker turns the lights off again.
Meanwhile, the people outside the building are asking a different question. Under NIS2, an organisation in scope owes its regulator an incident notification within 72 hours, including an initial assessment of severity and impact. Insurers want to know what happened before they decide what they will pay for. Neither will be satisfied by “we’re restoring”. They want to know the organisation is in control, and that is a question about assurance.
Two ways to fail
There are two ways to get cyber recovery wrong, and they sit at opposite ends of the same dial.
The first is the hospital’s first team: speed without assurance. Recover everything as fast as possible and hope the backups are clean. It feels decisive, it looks good for the first 48 hours, and it fails in a way that compounds.
The second is assurance without speed, and it usually comes from good intentions. Digital Forensics and Incident Response teams, quite rightly, want to understand what happened before anything goes back into production. Taken to the extreme, nothing is restored until the investigation is complete, every system is examined by hand, and every question has an answer. In an estate of thousands of workloads and a handful of forensic specialists, that queue is measured in weeks. The business can run out of cash, customers and patience long before the investigators run out of questions.
The organisations that come through well don’t pick one point on that dial and hope for the best. They decide in advance where each part of the business should sit on it, and they build a way of working that moves through recovery quickly without skipping the checks that matter.
Industrialise it, and build the trust into the line
Which leaves the question a board actually needs answered: how do we recover at the pace the business needs, at the scale of a modern estate, and still show that what we’ve brought back can be trusted?
Watch a Formula One pit stop. Four wheels are changed in little more than two seconds. It is fast because every person has one job, has rehearsed it until it is automatic, and works in parallel with everyone else. And the car is released only when all four wheels are confirmed secure. Nobody on that crew treats speed and safety as opposing forces. The safety check is part of the stop, and the sport’s rules make sure it stays that way: release a car unsafely and the penalty wipes out whatever time the crew saved. The stop time that counts already has the check built in.
Cyber recovery at scale needs the same idea: a production line with defined stations, specialist roles, automation where it helps, and quality gates every system must pass before it is released. Five elements make the difference.
Decide the running order before the incident. Not everything needs to come back at once, and not everything deserves the same scrutiny. Start from the Minimum Viable Company (or Minimum Viable Entity, for public sector and not-for-profit organisations): the smallest set of business services the organisation needs to survive, and everything those services quietly depend on. Traditional impact analyses tend to miss the dependencies that matter most in a cyberattack: identity, internet access, physical access control, and the regulated communications you owe to regulators, insurers, law enforcement and the people whose data was affected. Miss those, and someone will bring the identity system (usually Active Directory) back early just to get the doors open and the emails out. The attacker comes back with it.
Build stations with exit criteria agreed in advance. Each system moves through the same sequence: isolate, investigate, clean or rebuild, validate, release. Each station has criteria for leaving it, written and agreed before the incident, while people are calm. A system might only leave validation once it has been restored from a point before the earliest known attacker activity, checked for the indicators of compromise found so far, had known persistence removed, had the vulnerabilities used in the attack closed, and had its credentials reset. Your criteria will differ. What matters is that they exist before the crisis, so the 3am decision becomes a check against a standard the organisation already agreed.
Let machines do the checking and people do the deciding. Most of the work at each station is repeatable: scanning for known indicators, comparing configurations against a known-good baseline, confirming patch levels, checking that a restored workload behaves as it should. That work should be automated, because automation lets the line run at the speed of the business rather than the speed of the forensic team’s calendar. The judgement call is different. Whether the evidence is good enough to release a system before the investigation has finished is a decision for a named person, and it belongs to the business, supported by Security Operations and IT Operations.
Set the assurance level for each tier, and let the business own it. Some systems should be rebuilt from known-good sources and never restored. Identity infrastructure is the obvious example: if the attacker owned it, a restored copy hands it straight back to them. Other systems can be recovered and then remediated, provided they are watched closely afterwards. Choosing between rebuilding to a known-good state and recovering followed by threat remediation is a trade-off between speed and assurance, which makes it a business risk decision. Take it per tier, in advance, at the board or an executive risk committee, and write it down. On the day, the recovery team is then carrying out a decision the organisation has already made.
Make the evidence a by-product. Every gate a system passes through produces a record: what was checked, what was found, who released it and on what basis. Captured as you go, it costs almost nothing. Reconstructed afterwards, it is expensive and unconvincing. That record lets a board show regulators, insurers and shareholders that the organisation recovered in a controlled way, and it is what directors will be judged on long after the systems are back.
Put those five elements together and recovery becomes fast because it is organised, and trustworthy because the checks are built into the line. That is what I mean by industrialised recovery, with trust.
What the board should measure
If time to restore is the only recovery number in the board pack, the organisation will keep optimising for it. Four measures give a more honest picture, and each one can be tested in an exercise long before it is needed for real.
Time to trusted service. The time from declaring an incident to each Minimum Viable Company service running in production on systems that have passed their gates. It moves the question from “how fast can we restore?” to “how fast can we operate again, safely?” It is the number the hospital needed, and the one its status reports never showed.
First-time-right rate. Manufacturers call it first-pass yield: the share of output that passes inspection without rework. In recovery, it is the proportion of systems released into production that stay clean. A falling rate tells you the line is moving faster than its checks, and every failure is a system you will be recovering twice.
Throughput. How many systems the line can move through its gates in a day, with the people, automation and isolated capacity you actually have. Few organisations know this number. Measuring it in an exercise is the first honest test of whether your recovery time objectives were ever achievable.
Decision evidence. The proportion of release decisions with a named owner and a recorded basis. This is the measure that protects directors. It shows that recovery was governed, and it gives you an answer when a regulator asks how you knew it was safe to reconnect.
Four actions for your next board meeting
Ask for the Minimum Viable Company to be defined and agreed, including its dependencies: identity, connectivity, physical access and regulated communications.
Ask which systems will be rebuilt and which recovered and remediated, and make sure that choice is recorded as a business risk decision with an owner.
Ask to see the exit criteria for each stage of recovery, and who has the authority to release a system when the evidence is incomplete.
Add time to trusted service and first-time-right rate to the board’s resilience reporting, and ask for both to be measured in the next exercise.
Back to the hospital
The hospital had backups, technology and a team working flat out. Its first team restored quickly, over and over again. The missing piece was a way of recovering that treated trust as part of the job, and once the hospital had one, its systems came back and stayed back.
Speed still matters. A recovery that is perfectly assured and three months late is its own kind of failure. The aim is the right balance of speed and assurance for each part of the business: decided before the attack, built into the way recovery runs, and measured by the board in a way that rewards the outcome everyone actually wants. An organisation that is running again, and can prove it is safe to be running.
If you’d like to hear the full story of the hospital, I presented a session at Mandiant M-Wise alongside the hospital and Cisco Talos.
