How to run incident threads in chat

When production breaks, the first thing most engineers do is post in chat. That’s the right instinct: chat is where the people are, it’s fast, and it leaves a timestamped log. It’s also how incidents turn into chaos: forty people in one channel, three theories argued in parallel, the one message that mattered scrolled off-screen, and nobody sure who’s in charge.

The fix isn’t a different tool. It’s a small amount of structure, agreed before the incident, that everyone follows during it. This guide is that structure. It draws on the Google SRE book’s chapter on managing incidents, PagerDuty’s incident response docs and Atlassian’s incident management handbook, adapted to the way chat actually works. It applies to Slack, Teams, Discord, Zulip or Hasarak.

Decide these before the first incident

Severity levels

Severity decides who gets woken up and how loudly you communicate. Keep it to three or four levels, each defined by impact, not by cause:

Level	Meaning	Response
SEV1	Core product down or data at risk for many customers	Page on-call now, incident commander assigned, status page updated, updates every 30 min
SEV2	Major feature broken or seriously degraded, or a SEV1 for a small group	Page on-call, incident channel, updates every hour
SEV3	Minor feature broken, workaround exists	Incident channel or a thread, fix in working hours
SEV4 (optional)	Cosmetic or internal-only	A ticket, not an incident

This mirrors common practice: Atlassian uses SEV1–SEV3, PagerDuty SEV-1 to SEV-5. Two rules make severity work: anyone can declare an incident, and if you’re unsure, pick the higher severity; PagerDuty’s guide says the same, and to argue about it in the postmortem, not during the incident.

Roles

The single most useful idea from incident command systems is that the person fixing the problem should not also be coordinating it. Google’s SRE book separates incident command, operational work, communication and planning, and only the operations group changes the system; PagerDuty’s roles are Incident Commander, Deputy, Scribe, subject matter experts and liaisons. For most teams three roles are enough:

• Incident commander (IC). Owns the incident, not the fix. Decides severity, assigns work, keeps the channel on track, decides when it’s over. On a small team, the IC is whoever declared it, until they hand off.
• Operations lead. The engineers doing the investigating and fixing. Several people, often, each on one workstream.
• Communications lead. Writes the status updates for stakeholders and customers, so engineers aren’t interrupted with “any update?”.

Write the IC role down. The phrase “I’m IC for this incident” in the channel is a small ritual with a large effect: everyone knows where decisions come from.

Naming and where things go

• One channel per incident, named predictably: inc-2026-10-24-checkout-latency. Dates sort, keywords search.
• One standing channel for declarations (#incidents), where each incident gets one message linking its channel. That’s the index.
• Runbooks as docs that link from alerts, so the first responder doesn’t search for them.

During the incident

1. Declare it with a template

The first message in the incident channel sets the tone. Use the same template every time so people know where to look:

🔴 SEV2 declared: checkout p95 latency > 4 s since 14:05 UTC
IC: @maya
Impact: ~30% of checkouts timing out (EU and US)
Known so far: started after the 13:58 deploy of payments-api
Status page: updated, "Investigating"
Next update: 14:45 UTC
Workstreams: replies in threads below, one per hypothesis

2. One thread per workstream

This is the habit that keeps a busy channel readable. The channel is for decisions, status updates and the IC; each thread is one line of investigation:

#inc-2026-10-24-checkout-latency
├─ IC: Workstream A: roll back payments-api to 2026.10.24-1 (@sam)
│   └─ 6 replies: rollback started, canary green, full rollout…
├─ IC: Workstream B: is the DB the bottleneck? (@lee)
│   └─ 11 replies: connection pool graphs, slow query log…
└─ IC: 14:45 update: rollback done, latency recovering, monitoring

Rules that make it work:

• The IC opens each workstream as a top-level message naming the owner. Everyone else replies in threads.
• Findings go in the thread; conclusions go to the channel. When a workstream learns something that changes the plan, its owner posts one line to the channel.
• Nobody posts “+1, seeing this too” at the top level. Use a reaction.
• Side conversations move out. “Should we rewrite the pool?” is a follow-up, not part of the incident.

3. Status updates on a clock

Stakeholders stop interrupting engineers when updates arrive before they have to ask. Post them at the top level, on the schedule your severity promised, even when nothing has changed:

🟠 Update 14:45 UTC (SEV2, checkout latency)
Status: Mitigating
What we know: the 13:58 payments-api deploy added a synchronous fraud check
What we're doing: rollback done 14:38; latency back to normal in EU, US recovering
Customer impact: checkouts succeeding again; some retried payments may show twice as pending
Next update: 15:15 UTC or sooner if anything changes

“Nothing new” is a valid update. Silence isn’t.

4. Log decisions as they happen

Every decision gets one line in the channel, prefixed so it’s searchable later: DECISION: rolling back payments-api, not hotfixing (IC). These lines are the skeleton of the postmortem timeline.

5. Hand off explicitly

Long incidents outlast people. A handoff is a top-level message, not a quiet disappearance: “Handing IC to @lee. State: mitigated, monitoring. Open workstreams: B (DB pool). Next update due 16:15.” The new IC replies to confirm.

6. Close it

The IC declares the end: “Resolved at 15:40 UTC. Monitoring for 30 min, then closing. Postmortem owner: @sam, due Friday.” Then, and only then, the channel goes quiet.

The lifecycle, as a diagram you can put at the top of the runbook (Mermaid source in the diagrams guide):

After the incident: the postmortem

If you followed the steps above, most of the postmortem is already written: the declaration has the impact, the DECISION: lines are the timeline, and each workstream thread holds the evidence. A blameless postmortem focuses on how the system and process allowed the failure, not who typed the command.

A template that fits on one page:

# Postmortem: checkout latency, 2026-10-24 (SEV2)

**Duration:** 14:05–15:40 UTC (1 h 35 min) · **IC:** Maya · **Author:** Sam

## Impact
Who was affected, how, for how long. Numbers if you have them.

## Timeline (UTC)
- 13:58 payments-api 2026.10.24-2 deployed
- 14:05 p95 alert fires; SEV2 declared
- 14:20 DECISION: roll back, don't hotfix
- 14:38 rollback complete
- 15:40 resolved

## Root cause and contributing factors
## What went well
## What went badly
## Action items
| Action | Owner | Due | Ticket |

Write action items as tickets with owners and dates, or they won’t happen. Review the postmortem in a meeting within a week, while people remember.

Automate the boring parts

Most of the structure above can be helped along by bots and agents:

• Create the channel and post the template from a slash command or bot, so declaring an incident is one step.
• Post deploy and alert events into the incident channel from CI or your monitoring tool, so the timeline has machine timestamps.
• Remind the IC when the next status update is due.
• Draft the postmortem from the channel: the declaration, every DECISION: line and every top-level update, in order.

On Slack, dedicated incident tools do most of this out of the box: incident.io creates an #inc-… channel from /inc, Rootly creates a channel per incident, and FireHydrant builds the timeline from the channel’s messages. If you’re on Slack and run incidents often, they’re worth evaluating before building your own.

Doing it in Hasarak

Hasarak doesn’t have a dedicated incident product, and none of the tools above integrate with it. What it has is the building blocks, all scriptable:

• Channels and threads for the structure above, and an inbox that collects mentions, thread replies and DMs, so being pulled into a workstream doesn’t depend on watching the channel.
• Pages for runbooks and the postmortem, next to the incident channel, with Mermaid diagrams. See chat with docs.
• GitHub and Linear connectors that reply with a PR’s or issue’s state when one is linked, which helps when the fix is a PR. (Built, and coming soon to app.hasarak.com.)
• An MCP server in every workspace, so an agent can do the drafting.

That last one deserves an example. The workspace’s MCP server runs short TypeScript programs against the workspace as the member who signed in. This program, given to an MCP client’s execute tool, collects an incident channel’s declaration, decisions and updates and writes the postmortem skeleton as a page:

const channelId = "inc-2026-10-24-checkout-latency";
const lines: string[] = [];
let before: string | undefined;
do {
  const page = await tools.hasarak.messages.list({ channelId, limit: 200, before });
  for (const m of page.items) {
    const text = m.text ?? "";
    if (/^(🔴|🟠|DECISION:|Resolved|Handing IC)/.test(text)) {
      lines.push(`- ${m.createdAt.slice(11, 16)} ${text.split("\n")[0]}`);
    }
  }
  before = page.nextCursor;
} while (before);

const page = await tools.hasarak.pages.create({
  title: "Postmortem: checkout latency, 2026-10-24",
  body: ["## Timeline (UTC)", ...lines.reverse(), "", "## Impact", "", "## Root cause", "", "## Action items"].join("\n"),
});
return page.id;

In practice you’d ask the agent for it in plain words (“draft a postmortem from #inc-2026-10-24-checkout-latency”) and it writes a program like this itself, using tools.search() to find the tools. The MCP guide explains how that works and how to connect a client. Teams that want this as a button rather than a prompt can build a connector; connectors can also ship skills, written instructions such as a “triage an incident” checklist that @ai loads when a question in the chat calls for it.

The short version

1. Define severity by impact, and let anyone declare.
2. Name an incident commander who coordinates instead of fixing.
3. One channel per incident, one thread per workstream.
4. Status updates on a clock, at the top level, even when nothing changed.
5. DECISION: lines as you go; they become the timeline.
6. Hand off and close out loud.
7. Blameless postmortem within a week, with owned action items.

Further reading

• Google SRE book: Managing incidents and Postmortem culture
• PagerDuty incident response documentation
• Atlassian incident management handbook
• Hasarak for engineering teams