Skip to content

OT Incident Response: Plans, Playbooks and Recovery

Build an OT incident response plan around safety and process continuity: roles, decision rights, playbooks, evidence from PLCs and HMIs, and verified recovery.

Published on 8 min read

An incident in an industrial environment is not only a security event. It can be a production stop, an environmental release or a safety hazard, and the people who must act first are often operators in a control room rather than analysts in a security operations center. IT incident response playbooks that start with "isolate the host and reimage it" can make things worse when the host is the HMI that operators use to keep a reactor within limits.

This guide covers how to adapt incident response to OT: who decides, what the playbooks should contain, how to collect evidence from controllers and HMIs, and how to recover with confidence. It builds on the general incident response life cycle in NIST SP 800-61 Rev. 3 and the OT specific guidance in NIST SP 800-82 Rev. 3, which lists developing an incident response capability and a recovery and restoration capability as parts of an OT cybersecurity program.

The priorities are different

Every OT plan should state its priorities explicitly, in order:

  1. Safety of people on site and in the surrounding community.
  2. Protection of the environment and equipment.
  3. Continuity or controlled shutdown of the process.
  4. Preservation of evidence and investigation.
  5. Restoration of normal operations.

The order matters because it settles arguments in the moment. If preserving a compromised HMI as evidence conflicts with operators needing a working view of the process, the plan already says which wins. For background on why availability and safety dominate OT decisions, see OT vs IT security.

Roles and decision rights

The most common failure in OT incidents is unclear authority. Write down, before anything happens:

  • Who can order a process shutdown for cyber reasons. Usually the operations manager or shift supervisor, not the security team. Security recommends, operations decides.
  • Who can disconnect OT from the enterprise network. Pre-approve this action and define the physical or logical point where it happens (typically the industrial DMZ firewalls described in OT network segmentation). Test that the process can run without enterprise connectivity: historian replication, recipe downloads, licensing servers and time sources often turn out to depend on it.
  • Who contacts vendors and integrators, and under which support contracts. Vendors may need to help analyze controller logic or confirm firmware integrity.
  • Who handles external reporting: regulators, sector bodies, national CERTs, insurers. In the United States, CISA accepts incident reports from critical infrastructure operators; in the EU, NIS2 sets notification duties for in-scope entities. Know which rules apply before an event.

A cross functional team works best: OT engineering, operations, IT security, safety, legal and communications, with named deputies for night shifts and holidays.

Playbooks that fit OT

Generic playbooks are too vague. Write short, scenario specific ones and keep them printed in the control room, because the systems that hold your wiki may be down. Useful starting scenarios:

Ransomware in the enterprise network

The most likely scenario for most organizations. The question is whether OT keeps running.

  • Decide quickly whether to sever the IT/OT connection. A precautionary separation is often the right call; several operators have stopped production after enterprise ransomware because they could not be sure OT was unaffected or because business systems needed for operations were down.
  • Check the IDMZ: jump hosts, historian replicas, patch servers and file transfer shares are the usual bridges.
  • Hunt in OT for the same indicators before reconnecting.

Suspicious remote access session

  • Terminate the session at the jump host and close the conduit window (see secure remote access for OT vendors).
  • Identify which assets were reached and what engineering actions were performed.
  • Compare the logic and configuration of every touched controller against the known good baseline.

Unexpected change in controller logic or setpoints

  • Treat it as a potential safety issue first. Operations verifies the process is in a safe state.
  • Upload the current program from the controller and compare it with the stored golden copy.
  • Determine whether the change came from a legitimate engineering workstation, a vendor session or an unknown source.

Loss of view or loss of control

  • Operators fall back on manual operations procedures and local panels.
  • Distinguish a network or equipment fault from malicious activity. Many "cyber incidents" are failed switches, and many attacks first look like failures.

Each playbook should list the first actions, the people to call, the evidence to collect, and the criteria for escalating to a full incident.

Evidence collection in OT

Volatile evidence matters as much in OT as in IT, but collection has to be safe for the process. Plan your sources ahead of time:

Network traffic. A passive sensor on SPAN ports or taps is the most valuable OT evidence source, because it records what was actually sent to controllers. If you do not already run one, start a capture during the incident from an existing SPAN port with a ring buffer so you do not fill the disk:

tcpdump -i eth1 -n -s 0 -G 3600 -W 48 -w /data/ot-%Y%m%d-%H%M.pcap

Zeek with OT protocol parsers (for example CISA's ICSNPP plugins) turns those captures into searchable logs of Modbus, DNP3, S7comm, EtherNet/IP and other protocol activity. The OT asset inventory guide shows how to deploy this before you need it.

Controllers. Do not reboot or reflash a controller before capturing what you can:

  • Current program (upload from the controller using the vendor engineering software, from a trusted workstation).
  • Controller mode (run, program, remote), firmware version, and diagnostic buffer or event log, which often records mode changes, downloads and errors with timestamps.
  • Key switch position, if the device has one.

Engineering workstations and HMIs. These are usually Windows systems. Collect memory and triage artifacts if the process allows taking the host offline briefly; otherwise collect logs and project files live. Typical sources: Windows Security and System event logs, remote access tool logs, engineering software project files and their timestamps, and USB device history.

# Live hash and copy of engineering project files for later comparison
Get-ChildItem D:\Projects -Recurse -File |
  Get-FileHash -Algorithm SHA256 |
  Export-Csv \\evidence\ews01-projects-sha256.csv -NoTypeInformation

Historians and alarm logs. Process data shows what physically happened and when. Alarm and event logs often reveal setpoint changes or suppressed alarms that align with network activity.

Timestamps. Record time offsets between controllers, HMIs and the network sensor at the start of the investigation. OT devices often drift or run without NTP, and correlating events without offsets is painful.

Mapping what you find

Describing adversary behavior in a shared vocabulary helps when coordinating with vendors, CERTs and sector peers. MITRE ATT&CK for ICS gives you that vocabulary: for example Program Download (T0843), Modify Parameter (T0836), Alarm Suppression (T0878) or Denial of View (T0815). See MITRE ATT&CK for ICS: a practical guide for using it during and after an incident.

Recovery you can trust

Recovery in OT means returning the process to a known good state, not just turning computers back on.

  • Golden images and backups. Keep current, offline copies of controller programs, configurations, firmware files, HMI projects and workstation images. Store hashes alongside them. CISA's CrashOverride alert recommends keeping backups of gold images, configurations and install media for exactly this reason.
  • Verify before restore. Confirm backups are clean (they predate the intrusion, hashes match, files scanned) before loading them into controllers.
  • Reflash where integrity is uncertain. If firmware compromise is possible, follow the vendor procedure for firmware verification or reinstallation.
  • Rebuild, do not clean. For compromised Windows hosts in OT, rebuild from known good images rather than trying to remove malware.
  • Staged restart. Bring systems back zone by zone, with monitoring in place, and validate process behavior with operations at each stage.
  • Rotate credentials. Change passwords on controllers, HMIs, jump hosts and service accounts that may have been exposed.

Practice

Plans that have never been exercised fail in predictable places: unknown phone numbers, an untested disconnect, a backup nobody can restore. Run tabletop exercises at least yearly with operations present. CISA publishes free tabletop exercise packages, including scenarios for industrial control systems, that you can adapt. After a real incident or an exercise, record lessons learned and update the playbooks, the inventory and the segmentation rules.

The ICS attack case studies are good material for exercises: the 2015 Ukraine attack, for example, combined remote breaker operations, wiped HMIs, corrupted serial to Ethernet firmware and disrupted UPS units, which together tested manual operations and restoration far more than detection.

References

Related articles

Use MITRE ATT&CK for ICS in practice: its 12 tactics, sub-techniques, assets and detections, with jq queries, Navigator layers and coverage mapping for OT.
Run OT vulnerability management that works: read CISA ICS advisories, pull CSAF data, check KEV, rank by exposure and consequence, then patch or compensate.
Design vendor remote access to OT you can approve, watch and revoke: brokered sessions, DMZ jump hosts, MFA, time limits and firewall rules to enforce it.