AO Activity Events: How We Made Agent Failures Queryable

Agent Orchestrator already knew a lot about what happened inside a session. The problem was that most of that knowledge died in memory, logs, or silent catch blocks. When an agent failed to notice a PR, missed a review comment, kept polling a stale runtime, or triggered a report watcher repeatedly, the RCA path was usually human archaeology.

The activity event logging work changed that. Ashish built the durable event layer in PR #1528. I consumed it in PR #1620 by wiring the first high-value lifecycle-manager failure paths into that layer. This post is the team handoff: why the architecture exists, how to use it, what PR #1620 actually emits, and the rules I want every follow-up PR to preserve.

The Problem

AO runs autonomous coding sessions across trackers, SCM providers, runtimes, agents, webhooks, and the dashboard. That makes normal logs insufficient. A log line can tell us something failed; it does not reliably answer which project, which session, which PR, which lifecycle state, whether the event was repeated, whether sensitive data was stripped, or whether another agent can query it later.

The goal was not to build a full observability platform. The goal was smaller and more useful for AO: create a local, queryable, sanitized RCA trail that humans and diagnostic agents can read after the fact.

Design principle: activity events are evidence, not telemetry noise. Every event should answer a real debugging question.

What Ashish Built First

PR #1528 gave us the platform layer. It is intentionally boring infrastructure: write events locally, sanitize the payload, keep the query surface stable, and make the write path best-effort so AO never changes behavior just because event storage is unavailable.

The most important invariant from the infra PR is simple: recordActivityEvent must be safe to call from critical paths. If event storage fails, the lifecycle continues. That is why PR #1620 could add events in sensitive state-machine code without turning observability into a new failure mode.

Why FTS And JSON Matter

During the planning around issue #1511, the key idea was that the consumer of these events is not always a person reading a table. Often it is another agent trying to diagnose a broken AO run. That agent does not start with the exact event kind. It starts with a symptom: PR not detected, review comments missing, terminal vanished, cleanup skipped, report watcher fired too many times.

That is why FTS search and a stable JSON envelope are not nice-to-haves. FTS lets a diagnostic agent search like a human. JSON lets it reliably extract session IDs, project IDs, PR URLs, lifecycle states, attempt counts, and reasons. Together they turn local events into a usable RCA substrate.

ao events list --session ta-7 --json
ao events search "report watcher triggered" --project agent-orchestrator
ao events stats --since 24h

What PR #1620 Added

PR #1620 made lifecycle-manager the first serious consumer of the activity-event layer. I intentionally chose lifecycle-manager because it sits on the highest-value RCA path: it decides whether sessions are detecting, implementing, reviewing, reacting, cleaning up, or stuck. When it loses context, the whole system becomes hard to explain.

The PR added 17 event kinds across 18 emit sites in packages/core/src/lifecycle-manager.ts. The events were grouped into four bundles so review could reason about each class of behavior separately.

The implementation was deliberately local to existing catch blocks, success observers, and state transitions. It did not move lifecycle mutations. It did not add new state-machine branches. It did not make event writes part of control flow.

A Good Event Looks Like This

A good AO activity event is compact, attributable, safe, and queryable. It names the source layer, uses a specific kind, includes enough identifiers for correlation, and summarizes the RCA value in one sentence.

recordActivityEvent({
  projectId,           // SQL column: project/repo boundary for filtering
  sessionId: session.id, // SQL column: reconstruct one agent run end to end
  source: "reaction", // SQL column: subsystem that observed the edge
  kind: "reaction.escalated", // SQL column 'type': stable machine contract
  level: "warn",      // SQL column: severity for triage and filtering
  summary: "Reaction escalated after reaching max attempts", // FTS-readable headline
  data: {             // JSON text column: event-specific RCA facts
    reaction: "send_to_agent", // which automated action was attempted
    reason: "max_attempts",    // stable grouping key; avoid prose here
    maxAttempts,               // configured threshold AO compared against
    attempts,                  // observed value that crossed the threshold
  },
});

The call is synchronous and best-effort. It does not decide lifecycle behavior. Internally it opens the local SQLite event DB if needed, sanitizes the summary and data, writes one row, updates the FTS index through SQLite triggers, and periodically prunes old rows. If the DB is unavailable, AO increments a dropped-event counter and continues.

This is deliberately a hybrid schema. The dimensions every event shares are normal SQL columns with indexes. The details that only make sense for one event kind live in data as sanitized JSON text. That keeps writes simple while preserving exact filters for the things we always query by: time, project, session, source, kind, and level.

The data object is not a dumping ground. It should contain the few fields that make the event useful later: what AO was trying to do, what rule or boundary it hit, the configured limit, the observed value, and enough IDs to join it with neighboring events. If a field is useful across many event kinds and people need to filter by it constantly, that is a signal to standardize it or promote it later.

Rules For Follow-Up PRs

• Start with the RCA question. Do not begin with where can I emit an event? Begin with what would we wish we knew during an incident?
• Prefer state edges and exceptional paths. Emit when AO learns something, changes state, fails an integration call, escalates, retries, gives up, or completes cleanup. Avoid per-poll heartbeat spam.
• Keep ownership clear. If core already owns a spawn failure, the API or CLI should not emit a duplicate event for the same failure unless it adds new context.
• Preserve behavior. Event writes must never decide lifecycle behavior. Treat them as best-effort evidence after the real action has happened or failed.
• Use one-shot guards for repeated conditions. Report watcher triggers, detecting escalation, and long-lived stuck states need activation or milestone events, not one event every loop.
• Sanitize by construction. Include PR URLs, branch names, project IDs, stage names, and reason codes. Avoid raw prompts, secrets, tokens, headers, and large blobs.
• Test both sides. Test that the event fires with the expected shape, and test that AO still behaves if recordActivityEvent throws.

How The Seven Rollout PRs Fit Together

After PR #1620 proved the consumer pattern in lifecycle-manager, I split the remaining work into narrow PRs so different owners can finish implementation and tests without stepping on each other. The split follows ownership boundaries, not random event kinds.

The rule for these PRs is the same as #1620: add evidence at the boundary where AO learns something important, keep the emit local to that boundary, and prove with tests that instrumentation cannot break the product path.

What I Would Improve Next

How To Review An Event PR

• Ask what question it answers. If the answer is just it logs that we were here, the event is probably too weak.
• Check source and kind ownership. Source should name the layer that owns the event, not necessarily the file that happens to emit it.
• Check payload size and safety. The payload should explain the event without carrying secrets, raw prompts, or full external responses.
• Check ordering. The emit should sit after the successful action or inside the exact catch that already handles the failure.
• Check duplicates. If another layer already emits the same failure, either remove one or make the second event add distinct context.
• Check tests. A test should fail if the event disappears or if instrumentation starts throwing into the product path.

The Mental Model I Want The Team To Share

AO is a state machine wrapped around unreliable external systems. SCM can fail, runtimes can disappear, agents can stop responding, webhooks can be malformed, config can be invalid, and cleanup can be deferred. Activity events are how we make those edges explicit.

The point is not to log more. The point is to make AO explain itself. When a user asks why did my session get stuck?, why did the PR not get detected?, why did AO notify me?, or why did cleanup not happen?, the answer should be recoverable from local events without asking someone to reproduce the whole run.

That is why this work is worth doing across the codebase. It turns AO from a system that performs actions into a system that can also narrate the evidence behind those actions.