Cyber Resiliency Board Briefing 9: The Recovery Capacity Problem: Why Recovery Is Often Limited by People, Not Technology
Executive Summary
When organisations assess cyber recovery capability, the conversation usually focuses around technology.
How quickly can we restore from backup? How much recovery infrastructure do we have? Can we recover into the cloud? How many systems can the platform restore concurrently? Do we have enough compute, storage and network capacity? These are important questions, but they frequently overlook one of the biggest constraints I see when assessing operational cyber resiliency: people.
The vast majority of organisations whose operational cyber recovery capability I assess remain heavily dependent upon tacit knowledge locked in the heads of a relatively small number of employees for at least some part of their response and recovery process.
They are the people who “actually know how the network is laid out”, understand why an undocumented firewall rule exists, know which part of a complex Active Directory federation authenticates a particular application, remember the sequence required to bring an old business system back online, or know which apparently minor service everything else unexpectedly depends upon.
During Business-as-Usual this can appear manageable. During a destructive cyberattack, when hundreds, thousands or potentially tens of thousands of systems need to be investigated, rebuilt, validated and recovered, often with large numbers being worked on in parallel, those individuals rapidly become the constraint.
The organisation may have petabytes of recoverable data and enormous technical restoration capacity, but if every recovery decision still needs to pass through three people who understand how the environment actually works, the effective recovery capacity of the organisation is three people.
This creates operational risk, key-person risk, governance risk and ultimately resilience risk.
The Board therefore needs to ask: “Do we have enough of the right human recovery capacity to use the technology we have bought at the scale at which we may need it?”
Recovery Technology Scales More Easily Than Human Expertise
Modern infrastructure can operate at enormous scale. Backup platforms can restore many workloads concurrently. Cloud environments can provision hundreds of systems. Infrastructure-as-Code can rebuild environments quickly. Automation platforms can execute thousands of tasks.
Human expertise does not scale in the same way. One network engineer who understands twenty years of accumulated routing decisions remains one person, as does the single application engineer who remembers an undocumented configuration dependency remains one person. Their working day does not become 200 hours long because 200 systems suddenly require their knowledge.
In destructive cyberattacks, the restoration process is rarely as simple as the Restore… Start… Finished workflow assumed in traditional Business Continuity and Disaster Recovery scenarios.
Post destructive cyberattack, the recovery to a trusted state requires:
Determining the appropriate recovery point.
Understanding application and infrastructure dependencies.
Rebuilding or validating identity.
Establishing trusted administrative access.
Remediating vulnerabilities and attack surface.
Removing attacker persistence from configurations, binaries, libraries, email, registries, policy objects and networks.
Rotating credentials, secrets, certificates and keys.
Re-establishing integrations.
Implementing improved security controls and increased monitoring to prevent reattack
Validating remediations.
Performing technical and business testing.
Making a risk decision on whether a system is sufficiently trusted to repatriate to production.
Each step requires specialist knowledge. The consequence is that theoretical technical recovery capacity can be substantially greater than actual operational recovery capacity.
You may be able to restore 500 virtual machines simultaneously, but that does not mean your organisation can recover 500 virtual machines simultaneously to a state that won’t quickly be reattacked or reinfected.
The Keystone Employee Problem
Most complex organisations contain what I think of as keystone employees.
They may not always be the most senior individuals. Sometimes they are an architect who designed part of the environment fifteen years ago. Sometimes they are a senior engineer who has accumulated knowledge through years of troubleshooting. Sometimes they are the person everyone calls when a particular application breaks because nobody else quite understands how it works.
Remove that individual and a surprisingly large amount of organisational knowledge disappears with them. During a major destructive cyberattack, demand for these individuals can become extraordinary: identity, network, application recovery and incident response teams all need them. At the same time, executives want answers, often from the very same people who are already trying to keep multiple recovery workstreams moving.
Every recovery workstream suddenly has the same handful of people as dependencies. They stop performing work and start becoming queues. This is where recovery capacity starts to collapse: twenty teams are waiting on one network architect, dozens of application recoveries depend on the same Active Directory specialist, and hundreds of reconnection decisions are queuing behind a single security architect.
The individual may be working exceptionally hard, but the organisation's recovery throughput remains constrained by their ability to answer one question at a time. This is one of the reasons heroic recovery does not scale.
Keystone Employees Can and Will Break
There is another aspect of this that organisations are often uncomfortable discussing, people have limits.
A serious destructive cyberattack may require recovery operations to continue for days, weeks or months. The people carrying the greatest amount of organisational knowledge are often subjected to the greatest pressure.
They may be working excessively long hours while fielding constant calls and messages from multiple senior stakeholders, all while making decisions with significant consequences for customers, employees and, in some sectors, public safety. At the same time, they are operating under sustained uncertainty and fatigue, knowing that one poor decision could result in reinfection, reattack or a materially longer outage.
At some point the pressure becomes too much. In several major incidents I have been involved in, I have seen critical people simply quit during the incident. When they leave, the organisation does not simply lose an employee. It can lose undocumented knowledge that is actively required for recovery.
That is an extraordinary operational resilience risk.
If the recovery of an important business service depends upon one person continuing to work despite extreme pressure, the organisation does not have a recovery process; it has a key-person concentration risk.
Fatigue Changes the Quality of Decisions
Human capacity is not simply about the number of people available, it is also about the quality of the decisions those people can continue making. Recovery teams working sustained sixteen or eighteen-hour days do not remain equally effective indefinitely.
Fatigue increases mistakes. Context becomes harder to retain. Handoffs deteriorate. People repeat investigations that another team has already completed. Decisions are made without all the available evidence. Configuration mistakes creep into rebuilt environments.
This becomes particularly dangerous when recovery requires a large volume of risk decisions to be made quickly: whether a system is safe to reconnect, whether enough investigation has been completed, whether the vulnerability that enabled the attack has actually been remediated, whether an identity can be trusted again, or whether the organisation is prepared to temporarily operate with reduced controls and accept the associated residual risk.
The organisation therefore needs a sustainable recovery operating model, not a plan based upon critical personnel remaining awake until the incident ends. Organisations also need a way of recognising when fatigue and sustained pressure are beginning to degrade an individual's judgement, rather than assuming that someone will recognise it themselves. This requires active oversight: watching for changes in decision quality, repeated mistakes, unusually risk-tolerant or risk-averse decisions, difficulty retaining context, or simply an individual becoming a bottleneck because they are trying to carry too much.
Recovery leadership should have the authority to intervene, add additional expertise, introduce peer review for higher-risk decisions, rotate people out of critical roles and, importantly, give them sufficient protected time to sleep and recover. Taking a critical engineer or decision-maker offline for eight hours may feel counterintuitive in the middle of a major incident, but allowing an exhausted individual to make a decision that causes reinfection or forces part of the environment to be rebuilt again can cost the organisation days.
Cyber recovery is a marathon performed at sprint intensity. Unless staffing, shifts and handovers have been designed in advance, exhaustion eventually becomes another source of operational risk.
Tacit Knowledge Prevents Automation
The human bottleneck also limits one of the most important opportunities for increasing recovery capacity: automation. You cannot automate a process that the organisation cannot describe.
If when asking how you would rebuilding a specific service an engineer replies “I normally change these three settings, then restart that service, unless it is connected through the old authentication environment, in which case you need to speak to Dave…”, you do not yet have an automation opportunity. You have undocumented organisational knowledge.
Before a process can be orchestrated repeatedly, the underlying information needs to be externalised, which may include::
Recovery sequences.
Infrastructure dependencies.
Identity relationships.
Network flows.
Configuration requirements.
Required certificates and secrets.
Recovery images.
Software versions.
Validation criteria.
Security requirements.
Escalation conditions.
Reconnection criteria.
Rollback procedures.
Once this information is documented and structured, deterministic parts of the recovery process can begin to be automated.
The objective is not to remove skilled people from cyber recovery, it is to stop consuming scarce skilled people on tasks that do not require their skilled judgement.
The experienced architect should deal with exceptions, not personally explain the same architecture to 50 recovery teams.
Documentation Changes the Economics of Expertise
This creates an important distinction between knowledge ownership and knowledge execution.
In an immature recovery model, the expert both knows what must be done and performs or supervises every instance of it. This does not scale.
A more mature organisation extracts the repeatable knowledge from the expert and turns it into:
Runbooks.
Dependency maps.
Configuration repositories.
Infrastructure-as-Code.
Automation.
Recovery workflows.
Decision trees.
Validation scripts.
Architectural diagrams.
Known-good artefacts.
Standard operating procedures.
This allows a much larger pool of trained people to execute recovery while scarce experts focus on complex exceptions, architectural decisions and situations where judgement genuinely matters.
In effect, documentation allows the organisation to multiply expertise without multiplying experts.
Recovery Staffing Should Be Designed Around Throughput
Another mistake I see is assuming that the Business-as-Usual organisational structure will naturally become the recovery structure.
During normal operations, one infrastructure team may support thousands of workloads because only a small percentage require intervention at any one time.
A destructive cyberattack changes that ratio dramatically, as every workload may require attention. Systems need triage, rebuilding, security validation, configuration, testing and reconnection at approximately the same time, while the integrity or availability of your administrative identities and tooling, networks and configurations can no longer be trusted.
The staffing model for cyber recovery therefore needs to be based upon recovery workload and required throughput, not simply the existing IT organisation chart.
If 5,000 workloads need to be recovered within two days:
How many systems does each recovery team need to process every day?
How many identity specialists does that require?
How many security validation teams?
How many network changes?
How many application owners?
How many business testers?
How many simultaneous decisions can the governance process support?
How many systems can be safely moved through the recovery pipeline before one stage becomes a bottleneck?
These are capacity-planning questions that very few organisations have modelled for a realistic destructive cyberattack, instead assuming that the Recovery Time Objectives established for simpler, traditional Business Continuity and Disaster Recovery scenarios can be achieved using the same staffing model. In reality, those RTOs often take no account of the additional investigation, validation and decision-making required during cyber recovery, and expectations with senior executives frequently need to be reset around what is actually achievable.
Recovery Needs Cells, Not Heroes
At scale, recovery increasingly needs to operate through repeatable recovery teams or what I term recovery cells.
A recovery cell might combine people capable of performing:
Infrastructure recovery.
Security validation.
Identity integration.
Application configuration.
Business testing.
Evidence capture.
Reconnection.
Instead of a single central team touching every workload, multiple cells can operate in parallel against predefined recovery procedures. Specialists then become shared escalation resources rather than mandatory participants in every recovery.
This is similar to scaling any other complex operational process. You do not increase the output of a factory by having every item personally inspected and assembled by the chief engineer. You capture the chief engineer's knowledge in the process, tooling, quality controls and escalation criteria.
Cyber recovery needs the same discipline and scalability.
Incident Memory Is a Governance Asset
Documentation is also critical for another reason. A destructive cyberattack produces hundreds or thousands of decisions made by individuals.
Why was this recovery point selected?
Why was this system reconnected?
Why was an application permitted to operate temporarily without a particular control?
Why was one business service prioritised over another?
What evidence supported the conclusion that attacker persistence had been removed?
Why did the organisation believe an environment had reached a sufficiently trusted state?
During the incident these decisions may seem unforgettable, six months later they often are not. People forget, teams change, employees leave and external responders finish their engagement.
Once the pressure has subsided there is also an entirely understandable desire to put the incident behind the organisation and return to normality.
A regulator, cyber insurer, auditor or litigation process will operate on a very different timeline. Months later the organisation may be asked “What did you know at this point, what decision did you make, and what evidence did you use to support that decision?”
A collection of people's memories is not a governance framework. The organisation should therefore preserve contemporaneous:
Decision logs.
Evidence relied upon.
Risk acceptances.
Recovery approvals.
Investigation findings.
Validation results.
Exceptions.
Executive decisions.
Reconnection decisions.
Relevant communications.
This is the evidence that the organisation made controlled, rational and risk-informed decisions under difficult circumstances.
Memories fade. Governance obligations do not.
External Responders Cannot Read Your Mind
Most organisations will also depend upon external expertise during a major destructive cyberattack, including:
A retained Digital Forensics and Incident Response provider.
Specialist recovery resources.
A cyber insurer-appointed incident response team.
Specialist identity responders.
Recovery consultants.
Cloud providers.
Technology vendors.
Legal counsel.
Crisis communications providers.
I run teams that support organisations through exactly these situations, and one of the biggest determinants of how rapidly external responders can become effective is how quickly the organisation can explain its environment, decision-making structures and operational workflows.
Where are the critical systems?
How is identity structured?
Which networks are trusted?
Which systems constitute the Minimum Viable Company?
Which domain authenticates which service?
Which management plane controls this infrastructure?
Where are the relevant logs?
What am I authorised to do, and when and how do I escalate?
What are the critical third-party connections?
Who owns each service?
What must be recovered first?
If this information exists only in people's heads, external responders spend valuable time interviewing employees and reconstructing knowledge before they can begin solving the problem. This delays the value they can add while simultaneously consuming the time of the very internal resources they were brought in to help scale.
Good documentation therefore reduces what I term the incident response start-up latency. The faster responders can understand the environment, the faster they can begin producing useful results.
Staff Turnover Is Also a Resilience Problem
The value of capturing tacit knowledge exists long before an incident occurs. Cybersecurity, infrastructure and technology functions can experience significant personnel turnover, and every departing employee potentially takes some institutional knowledge with them.
New staff then spend months piecing together the knowledge that just walked out of the organisation. Good recovery documentation therefore also supports succession, cross-training and onboarding.
It converts organisational knowledge from something borrowed from employees into an institutional capability retained by the organisation.
The Recovery Knowledge Must Survive the Attack
There is, however, little value in creating excellent recovery documentation if it disappears during the incident.
This happens surprisingly easily when recovery plans sit in the corporate collaboration platform, architecture diagrams require production identity to access, passwords are stored in the same corporate password manager, contact lists are buried in email and runbooks can only be reached from endpoints that may themselves be compromised. At exactly the point that information is needed most, the identity service required to access it may have been disabled because it can no longer be trusted.
The organisation technically has excellent documentation, but operationally, it has none. Recovery knowledge should therefore be treated as part of the recovery control plane.
Critical recovery artefacts should be maintained in an environment appropriately isolated from production and accessible through a separate identity plane.
Depending upon the organisation, this may include:
Recovery plans.
Network diagrams.
Identity architecture.
Application dependency maps.
Minimum Viable Company definitions.
Emergency contact information.
Supplier escalation paths.
Recovery credentials.
Known-good configuration.
Scripts and automation.
Infrastructure-as-Code.
Software and licence information.
Decision templates.
Incident management procedures.
Regulatory and cyber insurance information.
Communication procedures.
The environment itself should have appropriate resilience, access control, integrity protection and auditability. There should also be a practical mechanism for accessing essential information if normal endpoints, identity, collaboration tools or networks are unavailable.
A recovery plan that cannot be accessed during recovery is simply documentation about a capability you no longer possess.
Reduce the Number of Decisions Required During the Crisis
Human recovery capacity can also be improved by reducing unnecessary decision-making. Every decision consumes time and cognitive capacity. During an incident, organisations repeatedly ask questions that could have been answered beforehand.
Who has authority to isolate this system?
Can this application operate without a particular security control?
Which service gets recovered first?
What evidence is required before reconnection?
Who accepts residual risk?
When do we escalate to the Board?
Which systems constitute the Minimum Viable Company?
Which third parties need to be contacted?
Wherever practical, these decisions should already have frameworks, thresholds, decision matrices, delegated authority and pre-agreed criteria.
The objective is not to eliminate judgement, it is to reserve human judgement for situations where judgement genuinely adds value.
Every decision safely made before the incident is one less decision competing for scarce executive and technical attention during it.
What Good Looks Like
Cyber resilient organisations deliberately manage human recovery capacity as part of their overall recovery capability. They:
Identify individuals and teams whose tacit knowledge creates recovery key-person dependencies.
Map those human dependencies into their Minimum Viable Company and critical business services.
Identify where one person, one team or one approval process could constrain large-scale recovery.
Capture critical tacit knowledge in documented and maintainable recovery procedures.
Convert repeatable recovery tasks into automation and orchestration wherever practical.
Build recovery runbooks that can be executed by competent staff without requiring the original subject-matter expert for every instance.
Maintain primary, secondary and tertiary resources for critical recovery roles.
Cross-train personnel rather than allowing knowledge to remain concentrated in one individual.
Exercise scenarios where key employees are deliberately made unavailable.
Design recovery teams around the required throughput of the recovery, not simply the Business-as-Usual organisational structure.
Establish recovery cells capable of operating multiple recovery workstreams in parallel.
Define sustainable shift patterns, handover procedures and escalation mechanisms for prolonged incidents.
Protect scarce specialists from becoming communication and approval bottlenecks.
Maintain structured decision logs that record what was known, what was decided and what evidence supported the decision.
Preserve investigation and recovery evidence for subsequent regulatory, insurance, audit or legal review.
Maintain an emergency information package that allows external responders to rapidly understand critical elements of the environment.
Store critical recovery knowledge outside the production trust boundary, using separate identity wherever practical.
Ensure essential recovery information remains accessible if normal endpoints, email, collaboration platforms or production identity are unavailable.
Measure recovery throughput and identify where queues form during exercises.
Most importantly, they recognise that people are part of the recovery architecture.
Human dependencies should be identified, protected, tested and engineered with the same seriousness as technical dependencies.
A Practical Analogy
Imagine a manufacturer invests in an advanced factory designed to produce 10,000 units a day, with the machinery, raw materials and automation all in place to support that level of output. On paper, everything suggests the factory has enormous production capacity.
There is just one problem. At six stages in the manufacturing process, production stops until one particular engineer arrives and explains what setting needs to be used. Every item moving through the factory eventually queues behind that engineer.
The manufacturer does not have a 10,000-unit-per-day factory, it has a factory whose capacity is determined by how quickly one engineer can answer questions.
The solution is not to ask the engineer to work faster. It is to capture their knowledge in the production process, automate repeatable decisions, train other people, establish quality controls and only involve the engineer when something genuinely exceptional occurs.
Large-scale cyber recovery works in much the same way.
The recovery platform may theoretically restore thousands of systems, but if every workload ultimately queues behind the same small group of experts, your recovery capacity is determined by the queue, not the platform.
Questions to Ask Your Executive Team
Which individuals possess knowledge without which recovery of an important business service would materially slow or stop?
Have we mapped those human dependencies into our critical business services and Minimum Viable Company?
How many of our recovery procedures rely upon tacit knowledge rather than documented processes?
If our most experienced identity, network, infrastructure or application specialists were unavailable tomorrow, could recovery continue?
Have we deliberately tested that scenario?
How many systems can our recovery teams realistically investigate, rebuild, validate and reconnect per day?
Is that recovery throughput sufficient for the size of our technology estate and our required recovery objectives?
Which stages of our recovery process are most likely to create queues?
Which specialist employees would be required by multiple recovery workstreams simultaneously?
Are those specialists performing activities that could instead be documented, delegated or automated?
Do we have trained backup personnel for every critical recovery role?
Can our staffing model sustain 24×7 recovery for multiple weeks without depending upon individuals working unsafe or unsustainable hours?
Do we have defined shift handovers and a reliable way to transfer context between teams?
Are recovery decisions and the evidence supporting them captured contemporaneously?
Could we demonstrate six months after an incident why a particular recovery or reconnection decision was made?
How quickly could an external incident response team understand our identity, network, application and business-service architecture?
Is the information they would require already prepared and accessible?
Where is our critical recovery documentation stored?
Could we still access it if our production identity service, collaboration platform, endpoints or network were compromised?
Does our recovery knowledge repository use an identity plane sufficiently separated from production?
How much of our current recovery process could be automated if the underlying knowledge were properly documented?
Key Takeaways for the Board
Cyber recovery capacity is frequently constrained by people long before it is constrained by technology.
Tacit knowledge creates hidden single points of failure within the organisation.
A small number of keystone employees can become bottlenecks when hundreds or thousands of systems must be recovered in parallel.
Excessive dependence upon individual expertise creates operational, succession and key-person risk.
Critical employees can become exhausted or overwhelmed during prolonged incidents, and the loss of one individual can remove knowledge required for recovery.
Technical recovery capability and operational recovery throughput are not the same thing.
Documentation is a prerequisite for effectively scaling recovery through automation and orchestration.
Scarce experts should manage exceptions, not become mandatory dependencies for every recovery action.
Recovery staffing should be designed against the scale and throughput required during a destructive event, not simply mirror the Business-as-Usual organisation chart.
Sustainable shift patterns, effective handovers and fatigue management are part of cyber resilience.
Recovery decisions and their supporting evidence need to be preserved because regulatory, insurance and legal scrutiny may occur long after memories have faded.
Well-structured documentation allows external responders to become effective more quickly.
Capturing tacit knowledge improves succession, onboarding and resilience during normal staff turnover.
Recovery documentation must itself be resilient and accessible independently of the production systems and identities being recovered.
People are not simply users of the recovery capability. They are part of the recovery capability.
The Board's Role
The Board should ensure that management understands its human recovery capacity, not simply its technical recovery capability.
When presented with recovery metrics such as restore speed, backup throughput or cloud recovery capacity, the Board should challenge whether the organisation has enough trained people, documented knowledge, decision-making capacity and operational processes to safely use that technology at the required scale.
The Board should expect management to identify material key-person dependencies and demonstrate how those dependencies are being reduced through documentation, cross-training, succession planning, automation and realistic recovery exercises.
It should also challenge recovery assumptions that depend upon critical individuals remaining continuously available. An organisation that can recover only while a handful of exhausted experts remain in the room does not have a resilient recovery capability.
The Board should also ensure that recovery governance creates an enduring organisational record.
Months after the incident, management should still be able to explain:
What information was available.
What decisions were made.
Who had authority to make them.
What evidence supported those decisions.
What residual risks were accepted.
Why services were considered sufficiently trusted to resume operation.
Finally, the Board should satisfy itself that the knowledge required to recover the organisation would actually remain available during the incident. If the recovery plan, architecture, credentials and contact information all depend upon the same production identity and collaboration environment that is being recovered, the organisation has created another circular dependency.
The fundamental Board-level question is: “Can our organisation sustain the people, knowledge and decision-making capacity required to recover at the scale of the event we are planning for, and still achieve our recovery objectives?”
Closing Thought
Organisations spend enormous amounts of money increasing the technical capacity of their recovery platforms through faster storage, additional cloud capacity, greater levels of automation and concurrency, and increasingly sophisticated security technology.
Yet the effective speed of recovery can still be determined by one tired engineer on their eighteenth hour of a conference call because nobody else knows how a critical piece of the organisation works.
That is not a technology problem, it is a resilience design problem.
The objective should not be to eliminate expertise. Expertise will always be essential during a complex cyberattack. The objective is to prevent expertise becoming a bottleneck.
Capture the knowledge.
Preserve the evidence.
Cross-train the people.
Automate the repeatable.
Pre-authorise the predictable.
Protect scarce specialists for the decisions that genuinely require their judgement.
Build a recovery model that can continue operating when individual people become tired, unavailable or leave altogether.
You do not have scalable cyber recovery simply because your technology can operate at scale. Your recovery only scales when your people, processes, knowledge and decision-making can scale with it.
Next Briefing
Cyber Resiliency Board Briefing 10: Building the Recovery Factory: Why Enterprise Recovery Must Be Designed for Throughput, Not Heroics
