How we distill decisions from Slack noise

A busy engineering Slack channel produces hundreds of messages a day. On a good day, one of them is a decision. Not a status update, not a debate, not a joke about the deploy. An actual call: "ok let's lock it: JWT, 15 min. shipping it."

That message is worth more than the other several hundred combined, because it constrains everything built afterward. It is also indistinguishable from the noise around it to every tool the team uses. Slack scrolls it away. Search finds it only if you already know it exists. And no coding agent will ever read it before renaming the auth endpoint.

Lockstep's job is to catch that message, turn it into a rule agents actually receive, and drop everything else. This post is about the machinery that does the catching, the distillation funnel we built for the Slack pipeline, and about the two design commitments that make it trustworthy: cost that scales with candidates rather than messages, and a human gate that nothing can skip.

What counts as a decision

Everything downstream trusts the extraction: session-start briefings, PR gates, the ledger itself. So the first problem isn't connecting to Slack. It's definitional: given a firehose, what qualifies?

We use four tests. They're not editorial guidelines; they are the literal grading instructions given to the extraction model, and the definition human reviewers see in the queue.

Durability. Does it constrain future work beyond the task at hand? "Deploy's done, prod is green" fails; that's a status update. "All new services log in JSON, no exceptions" passes. A one-off action is a change or a task, not a decision. This mirrors the change-vs-decision split our capture hooks already make for git diffs, applied to human text.

Agreement. Was it actually concluded? "I think we should move to pnpm" fails. That's a proposal, maybe a good one, but filing it as a rule would be lying about the state of the team. "Agreed, pnpm from next sprint" passes. Threads that fail this test aren't discarded outright; a decision-shaped thread that never concluded gets filed as a question ("did the team decide this?"), never as a rule.

Agency. Was it a real choice among alternatives? "The API rate limit is 100 req/min" fails if it's someone stating a fact about a vendor. "We'll cap our own API at 100 req/min rather than queue" passes, because someone weighed options and picked. Facts belong in docs. Choices belong in the ledger.

Specificity. Can it be restated as one imperative sentence? "We should be more careful with migrations" fails: careful how, enforced where? "Every migration ships with a down script" passes. If it can't be one rule, it can't bind anything.

Fail any test and the thread is deliberately dropped. The rubric is aggressive on purpose. A decision ledger polluted with vibes and status updates is worse than no ledger, because agents will obey it.

The funnel: recall, then precision, then a human

Running a strong model over every message would be expensive and, worse, noisy. So the pipeline is a funnel that starts cheap and permissive and ends expensive and strict:

1. Segment: group messages into conversation units. For Slack, a thread is one unit; sweeps pick up from a per-channel watermark.
2. Recall: a cheap filter asks "could this unit contain a decision?"
3. Extract: one structured LLM call grades survivors against the four tests.
4. Scope: map the decision to a canonical code surface, if it names one.
5. Dedup: never re-distill a thread we've already seen.
6. Gate: confidence and finality decide: propose, raise a question, or discard.
7. File: a proposed decision lands in the review queue with full provenance.
8. Human: a person confirms, edits, merges, or rejects. Only now can it bind.

The recall stage is tuned to over-trigger: passing junk forward costs a little money at the next stage; missing a real decision costs the whole point of the product. Precision is added downstream, and the final precision gate is guaranteed, because it's a person.

Stage 1: cheap recall

The first pass costs nothing. A keyword prefilter looks for decision-shaped language: "let's go with", "we decided", "final call", "going forward", "from now on", "approved", "let's lock", "ADR". Threads with none of these markers are dropped without an API call, and that's most of them.

Survivors get a second, still-cheap check: a small, fast model (Haiku in our tiering) answering a single binary question: might this thread contain a durable, agreed decision? The prompt tells it to say yes when unsure. That's the whole job: be fast, be cheap, don't miss.

Stage 2: structured extraction

Candidates that survive recall get the expensive call: a stronger model (Sonnet) with the four-test rubric as its system prompt and a forced JSON schema as its output contract. It returns the rule as one imperative sentence, the rationale, the alternatives considered, who assented, when, a free-text scope hint, any code surfaces the thread mentions, a calibrated confidence score. And evidence.

Evidence is the field the whole design leans on. Every extraction must include one to three verbatim quotes from the thread, copied exactly, never paraphrased. An extraction with no quote is discarded before any human sees it. This does two jobs at once. It's the provenance trail: every rule in the ledger is anchored to the exact Slack messages that produced it. And it's the hallucination check: if the model invents a rule, it arrives holding a quote the reviewer can falsify in one glance. A fabricated decision with real-looking evidence is much harder to produce than a fabricated decision alone.

Borderline extractions (a decision claim with middling confidence) get one re-check from the strongest tier (Opus), and we keep whichever grading is more confident. The expensive model runs only on the cases the mid-tier model itself is unsure about.

The economics that make it viable

Model tiering is half the cost story. The other half is that the expensive stage is built to be cheap per call.

The rubric and schema never change between calls, so the system prompt is prompt-cached: on Anthropic's API, cache reads are billed at 0.1x the base input price, so every thread after the first reuses the rubric at a tenth of the cost. And because scheduled sweeps don't need interactive latency, the extraction stage can run through the Batch API, which halves the price again. Stack the funnel on top and the shape of the bill changes: cost scales with candidate volume, not message volume. A channel can be arbitrarily noisy; you only pay real money for the threads that might matter.

Scope: where a decision lands

A decision about POST /auth/session should reach the agents and people who touch POST /auth/session. So stage 3 tries to canonicalize any code surface the thread mentions into the same grammar our capture hooks extract from git diffs: http:POST /auth/session, proto:pkg.Service/Rpc, gql:Mutation.field. When it matches, the ingested decision inherits real blast-radius impact from the dependency graph, exactly like a decision an agent proposed from code.

Plenty of real decisions are about process, not endpoints. When the thread names no surface, we don't pretend. The decision lands on a topic scope with impact zero, honestly flagged as unscoped, ranked low in the queue. Guessing impact would be worse than admitting we don't know it yet.

Dedup and the one-decision-many-provenances problem

The shipped dedup is idempotency: every unit hashes to (connection, external id, content hash), and a thread we've already distilled is never distilled again. That's the correctness backstop under the sweep watermark: re-sweeping a channel can't mint duplicates.

Semantic dedup (recognizing that a Notion page and a Slack thread describe the same decision) is designed but not shipped. The design matters enough to state: when a new extraction matches an existing decision in the same scope, the right move is to attach it as another provenance on the existing decision, not to mint a twin. One decision, many sources. When a new extraction contradicts a binding decision, it files as a proposed supersession with both versions surfaced to the reviewer. And when the pipeline can't tell, it falls to a human merge in the queue. Ambiguity routes to people; it never resolves itself silently.

Nothing binds without a human

The last stage is a person, and I want to be precise about why, because "human in the loop" usually reads as a liability disclaimer. Here it's load-bearing architecture.

A confirmed decision in Lockstep has teeth. It shows up in every agent's session-start briefing. It can fail a pull request that contradicts it. An extraction error at this altitude is not a bad search result you can scroll past. It is a false rule that machines will enforce. So the pipeline is built so that no code path exists from Slack to binding. Extractions are filed as proposed. Confidence ranks the review queue and pre-selects the reviewer's default action; it never lets anything skip the queue. The gate isn't a threshold we might tune down later. There is nothing to tune: binding requires a human event, structurally.

The review itself is designed to take seconds, not minutes: the reviewer sees the one-sentence rule, the verbatim quote, a link back to the source thread, the suggested scope, and four buttons: confirm, edit, merge, reject. The evidence quote is why this is fast. You're not auditing a model's reasoning; you're checking a sentence against a quote.

And every one of those clicks is training data. Rejects and edits are written back as labeled eval examples, alongside a golden set of hand-labeled real threads that gates every prompt change on measured precision and recall. Reject rate and edit distance are the quality metrics that matter. Extraction quality is a tuning problem, and the review queue is the flywheel that tunes it. The humans are the safety gate. They are also the label supply.

What's shipped, plainly

The Slack pipeline described here is the shipped, verified path: allowlisted channels only (nothing is swept unless someone opts a channel in), scheduled sweeps, the full funnel, the review queue, and confirmed decisions flowing into the same briefings and gates as everything else in the ledger. Evidence quotes are stored; full raw thread content is not. Jira, Notion, and Confluence are designed to enter through the same connector interface with per-source segmentation and prompts, and semantic cross-source dedup comes after that. I'd rather say that plainly than imply a breadth we haven't earned yet.

The deeper reason this pipeline exists is the one I wrote about in the load-bearing hallway: decisions on real teams are made out loud, in threads, in the ambient layer that agents never attend. A decision log that depends on humans remembering to write entries rots. The distillation funnel is the alternative bet: let people decide where they already decide, in Slack, and make the machinery do the filing. The rubric decides what's worth keeping. The funnel makes it affordable. The human makes it true.