The first sign is rarely a siren. It is a support queue filling with the same complaint, or a colleague forwarding a screenshot of your product confidently stating something that is not true. Sometimes it is quieter still: someone from security asking why conversation logs are sitting in a public storage bucket.
What happens over the next few hours depends mostly on work you did weeks earlier. Teams that prepared contain the damage in an afternoon. Teams that did not spend that afternoon in a meeting about who is allowed to switch the feature off. Here is how to be the first kind of team, in five steps.
Start by agreeing on what counts as an incident
Not every strange output deserves a war room. If everything is an incident, nothing is. Write down the categories your team actually cares about, and what each one triggers:
- Harm. An output leads to a real loss — a wrong refund, a denied claim, a shipment that should not have gone out.
- Exposure. Data ends up somewhere it should not be: in a prompt log, a shared dashboard, a vendor's pipeline, or another customer's answer.
- Misuse. Someone uses the system to do what your policy forbids, whether that is an employee or a customer probing for weaknesses.
- Degradation. Quality drops after a deployment, a prompt change, or a silent update from a provider.
- Dependency failure. A model endpoint, retrieval index, or tool you call starts timing out or behaving differently.
Then attach a rough severity. Two lines is enough: S1 means people are being harmed or data is exposed right now, and someone gets woken up. S2 means something is clearly wrong and needs attention today. S3 means log it and look at it this week. The point is not precision. The point is that nobody has to debate definitions while a live problem is still running.
Step 1 — Draw the map while things are calm
An incident plan is only as good as the inventory behind it. You cannot shut down what you cannot name, and you cannot warn customers about a system nobody documented. Keep one page per system, and include:
- What it does, and which users see its output.
- The model, provider, and version, plus the person who can change them.
- What data goes in, and where prompts, outputs, and feedback are stored.
- Human checkpoints — where a person reviews, approves, or can override.
- How to turn it off, throttle it, or roll it back, and how long that takes.
Test that last item once, on a quiet Tuesday afternoon, before you need it. A kill switch nobody has pulled is a hope, not a control.
Step 2 — Give people names and real authority
For a small team, three roles are enough: an incident lead who runs the response and makes the call, a technical owner who knows the system, and a communicator who handles everyone outside the room. Add a privacy or legal contact for anything touching personal data, and name a deputy for each role. Incidents have a habit of starting on a Friday evening in August.
The most commonly skipped item is authority. Decide in advance who can disable a feature, and give at least two people that power. If approval has to climb three levels while a model is telling customers the wrong thing, your response time is not measured in minutes.
If you handle personal data or work in a regulated sector, notification duties can carry deadlines measured in hours. Confirm what applies to you with qualified legal counsel rather than guessing.
Step 3 — Make sure you would actually notice
Most incidents are found by a person who happened to look. That is a thin line of defence. Thicken it with a few cheap habits:
- One obvious place to report problems, checked daily by someone whose job it is.
- Alerts on error rates, refusal rates, latency, and unusual output length.
- A small set of automated checks that run against real traffic samples and page a human when scores fall.
- A weekly skim of a random sample of sessions, done by a person with judgment.
- An on-call rotation where the person on call knows they are on call.
When something looks off, triage with five questions: is it still happening, how many people are affected, is data involved, can we reproduce it, and what is the worst plausible reading of what we are seeing? That last question is the one that keeps small problems from becoming expensive ones.
Step 4 — Write the playbook: contain, assess, communicate, recover
Under pressure, people follow sequences, not principles. Keep this one visible and short.
- Contain. Stop the bleeding with the blunt option first: disable the feature, roll back the model or prompt version, throttle traffic, add a hard filter, rotate exposed keys, revoke a tool's access. You can always restore service later.
- Assess. Establish scope — how many sessions, whose data, when it started, whether it is ongoing. Write it down as you go, because someone will ask and nobody remembers accurately at hour four.
- Communicate. Give internal updates at fixed intervals in a simple shape: what we know, what we do not know, next update at four. Then brief customers, partners, or regulators as required — plain language, no defensiveness, a clear statement of what you are doing about it.
- Recover. Bring the system back gradually with tighter monitoring, and keep the rollback option open for at least a few days.
Step 5 — Rehearse, then fix what the rehearsal exposes
Run a ninety-minute tabletop once a quarter. Pick one scenario and follow it honestly: the summarisation feature has been pulling documents from a folder it should never have had access to, and it has been doing so for six days. Watch where the group hesitates. That hesitation is the gap.
Within a week, hold a review that focuses on systems rather than blame. People hide information when they expect punishment, and hidden information is how the next incident gets worse. Finish with three to five action items, each with a named owner and a date, and check them at your next planning meeting. A plan that nobody updates quietly becomes fiction.
Keep it short enough to use at two in the morning
Nobody reads a forty-page document during an outage. Put the essentials on the first page: who to call, how to shut the thing off, where the logs live, and who talks to customers. Everything else belongs in an appendix. Store the plan somewhere people can reach it when the internal wiki is having its own bad day.
Review it after every incident and after every significant change — a new model, a new data source, a new market. You will not get your first response perfect. You only need to be faster and calmer than you would have been yesterday.
Photo: 3844328 / Pixabay
