Operations: Diagnostics and Recovery#

Part of the operations guide: reading the artifacts a run produced, and the playbook for a task parked in manual_action_required.

6. Diagnostics — reading what a run produced#

All artifacts live under <repo>/.worc/logs/<task-id>/. SQLite (<repo>/.worc/state.db) is the authoritative state; the artifacts and ledger are the human-facing index.

logs/
  completed.jsonl                 # append-only ledger: one record per terminal transition
  <task-id>/
    task.normalized.json          # the parsed task
    validation_report.json        # §19 gate verdict (pass or reject reason)
    task.enriched.md              # refinement output (if it ran)
    plan.md                       # planning output
    current.diff                  # working-tree diff at the last checkpoint (redacted)
    change-overview.txt           # its bounded map: changed paths, kinds, line counts, base SHA
    publish-error.txt             # the redacted git stderr of a failed publish node
    hitl/*.json                   # durable redacted question/approval + recovery handles
    prompt-audit/                 # who-ran-what + redacted prompt (only when prompt_audit)
      <NNNNNN>-<node>[-subNN].md  #   one step: fenced-json header, then the prompt verbatim
      timeline.jsonl              #   the machine-readable half: one whole record per line
    summary.md / summary.json     # the whole-task handoff → PR body ({what, summary, [follow_ups], …})
    failure_report.json / stuck.md# written when a loop guard or budget stopped the task
    recovered-failure_report.json / recovered-stuck.md  # the same pair, renamed at `done`
    merge/conflicts.md            # merge-task's conflict inventory (only on a conflicted merge)
    checks/<NNN>.log              # each check command's output (redacted)
    stages/<node-id>/
      history.jsonl               # one line per run of this node (run_id, outcome, findings, dir)
      run-<node-run-id>/          # one dir per node run — every re-run kept, never clobbered
        rendered-prompt.md        # the exact node prompt sent (redacted; from the node's role_file)
        findings.json / summary.md# review/evaluator findings + summary (this pass)
        <node-id>.out.md          # the node's output, exposed downstream as {<node-id>_path}
        citation.json / dependency_scan.json  # checks-node reports (this pass)
        stdout.txt / stderr.txt   # tool-node redacted streams
        <attempt>-<provider>/     # provider attempts for this run
          request.json            # redacted request (argv, permission profile, context paths) — kept at every level
          stdout.log / stderr.log # redacted process output
          events.jsonl            # redacted provider event stream
          result.json             # normalized AgentRunResult, incl. effective model/reasoning
    publish/terminal-cleanup.json # the terminal base-branch cleanup decision
  • The ledger (completed.jsonl): grep here first — id, title, branch, pr_url, final_status, fix_iterations, terminal cleanup status, and a pointer to failure_report.json when stuck. A record of a run that did not finish also names where and why it stopped: stop_node, stuck_loop (null for an infrastructure or operator stop) and stuck_limit (max_fix_cycles, repeated_findings, …, or the first line of the stop reason) — taken from this run's own stop, not from a report path an earlier attempt left on the task row, and null on a done record or an older one. It holds one record per terminal transition — not per run and not per provider attempt: a task that failed, was rerun and then finished has two records, an interrupted run that was resumed has one. attempt is the record's ordinal among the id's terminals (gate refusals excluded), counted when it is appended. A done record with recovered_from_stuck: true and a recovered_loop is a task that finished although a loop guard had stopped one of its fix loops (below); read cost and duration from state.db, not from a record count.
  • Why a task is stuck: open stuck.md (human-readable) / failure_report.json (machine). They record which fix loop and which limit was exhausted, all counter values, the failing node's provider attempts, the last failing check output, the last blocking review findings (rendered in stuck.md as Markdown bullets — severity, gating, reason, paths, proposed fix), and the final diff — plus, for a decomposed task, the failing subtask k of n and the SHAs already committed. When the stuck node is a tool node, which never writes a check log, last_check_log_note says so and points at its stdout.txt instead of leaving a bare null. limit_exhausted names the limit — a budget (max_fix_cycles, max_total_fix_iterations, budget:<from>-><to>; a budget of N allows N fix rounds, so with max_fix_cycles: 15 fifteen rounds run and the sixteenth fail/rework is the one refused) or one of the two no-progress guards that end a loop before its budget: no_file_change (the agent kept producing output without editing the tree across two consecutive rework charges; in-memory, so it resets on resume) and repeated_findings (the same node returned an identical non-empty finding set three times in a row — the fixer kept working and the verdict never moved, which usually means the finding asks for something the node cannot change). The second is derived from the recorded verdicts of the current attempt, so a restart does not reset it and a fresh rerun does not inherit it. Neither guard, and no budget, ever parks a non-blocking evaluator (blocking: false): it accepts on whichever limit it reaches first — its own max_rework_per_stage or the budget of the loop its rework edge charges (or the global backstop) — with a console WARNING naming that limit (evaluator accepted after exhausting its rework budget (<limit>) — continuing; stage may need follow-up) and a ⚠️ trace, never manual_action_required. A task can still reach done after such a stop — continued by an operator, or past an evaluator that does not gate publication — and at the transition to done the pair is renamed recovered-failure_report.json / recovered-stuck.md, the task's pointer to it is cleared, and worc list shows done (recovered: <loop>).
  • Audit completeness: SQLite records every node_runs and provider_attempts row (primary and any fallback), each artifact is registered with a sha256 checksum, and every commit/push/PR carries an idempotency fingerprint so a restart never double-publishes. Every provider_attempts row carries a task_id, and the constant supervisor layer's own provider calls are recorded too (with node_run_id NULL, since it is not a flow node) — so a whole-task cost/usage roll-up is SELECT … FROM provider_attempts WHERE task_id = ? (no node_runs join) and includes the supervisor spend. Per-attempt cost (usage_cost) is filled from the provider's own figure when the CLI emits one — Claude's stream-json total_cost_usd, which covers the subagents the attempt ran, and its token counts are read from the same result's per-model modelUsage so they cover them too (a CLI that reports no modelUsage falls back to usage, the main agent's last turn only). An attempt the orchestrator rejected after it ran (e.g. background_work_lost) still records its usage — the money was spent. Codex emits no dollar figure, so its usage_cost stays NULL, and that gap is named rather than estimated: summary.json's cost_not_accounted, worc status's cost_not_accounted= line and the worc list table footer say how many attempts and input tokens carry no price. The recorded total sits beside it: summary.json's task_usage (every node attempt plus the supervisor layer, in total and per provider), worc status's cost= line and the PR body's ## Run cost section, all in one wording — $114.11 over 19 provider calls (Claude's reported cost includes its subagents), the note present only when Claude ran. Each row also records the effective model / reasoning the attempt ran on, so per-model spend is a GROUP BY model with prompt_audit off. artifacts.path is stored POSIX and relative to the worc home (the lifecycle summary reads ../tasks/done/<id>.summary.md), so state.db names no home directory. A hard --force-full kill SIGKILLs the daemon mid-node, so the running node's own completion write never happens; the next terminal transition (finalize, or a clean pipeline terminal) reconciles those orphans — the node_runs row is closed to aborted with its reason in abort_reason and the killed attempt gets a provider_attempts row with usage_delta_status='unknown' — so an operator-killed run is auditable, not silently stranded running. A node that stops before its provider call (a containment breach, an integrity failure, an unresolvable tool) closes its own row the same way on any path, the merge flow included, because the row's lifetime belongs to the node layer. skip_reason means only a when-false or disabled node (skipped = 1); it never carries an abort. Both audit tables record the attempt/check's real measured interval (started_at/finished_at taken from the run, not two clock reads at row-write time), so a duration roll-up is meaningful.
  • Node run vs. attempt: run-<node-run-id> is reserved in SQLite before the provider starts and changes for every repeated node invocation, including each fixing cycle and recovery run. <attempt> starts at 1 inside that run and increments only for provider fallback.
  • Per-run history is preserved, never overwritten: every operator-facing artifact a node produces (the rendered prompt, review findings.json/summary.md, the generic <node-id>.out.md, checks reports, tool streams) is written under that run's stages/<node-id>/run-<node-run-id>/ dir, next to its provider attempts — so a review → fixing → testing → review loop keeps each pass's findings instead of clobbering the last. Read stages/<node-id>/history.jsonl for the chronological index of a node's runs (one line per run: run_id, outcome, findings count, dir). This history is exempt from logging.artifacts pruning and is removed only by an explicit worc logs clean of the whole task tree.
  • No secrets anywhere: request.json, the stdout/stderr/events logs, diffs, SQLite rows, the ledger, and the failure report are all redacted; denied_read_paths (.env, secrets/**) are excluded from agent reads and their values are scrubbed from any sink.

  • The agent-facing exchange and its terminal snapshots (WRI-007): while a task runs, its curated agent-readable artifacts (plan, diff and its change-overview.txt, findings, checks input, HITL packet) live in the in-repo exchange <repo>/.worc-io/<task-id>/. At every terminal status (done/failed/manual_action_required) the orchestrator seals a checksum-verified copy into the private audit at .worc/runs/exchange-seals/<task-id>/seal-<NNNNNN>/ (manifest.json + the sealed files) and then removes the active .worc-io/<task-id>/ — so a terminal task never leaves an agent-readable exchange behind for the next task. Debug a terminal exchange from the newest seal-* snapshot (and the raw .worc/logs/<task-id>/ audit), not from .worc-io/. rerun --continue restores the latest verified snapshot before resuming; a fresh rerun/restart starts from an empty exchange. If mutation detection (WRI-002) flags an agent-side change, the tree is quarantined instead of sealed under .worc/runs/exchange-quarantine/<task-id>/<NNNNNN>/ (with evidence.json recording the expected vs observed manifests plus created_at, the mutating node_id and the task attempt; a bundle with nothing to record is not written) and named on the task row in quarantine_refs, which outlives the contamination flag. The daemon's own crash recovery refuses to resume over a contaminated exchange; an operator rerun --continue clears the flag and continues, logging where the kept bundles are. There is no configuration option that keeps a terminal exchange agent-readable. Seals are evicted automatically for a successful task (logging.clean_runs_on_success) and on demand with worc runs clean; quarantined evidence is removed only by runs clean --include-quarantine.

  • Rendered node prompts: stages/<node-id>/run-<node-run-id>/rendered-prompt.md is the exact (redacted) instruction the agent received for that run — read it first to confirm a role_file edit took effect and rendered as intended (a re-running node keeps one per run). This includes the constant supervisor layer's own turns (stages/supervisor/run-<node-run-id>/) — observe/finalize/handoff are audited the same way, even though the supervisor is not a flow node.

  • What a node was handed about the change: from the first workspace-write edit on, every current.diff is written together with change-overview.txt from the same snapshot — one line per changed path (A/M/D/R old → new, added/removed line counts; a security.denied_read_paths match listed by name only, [denied]), the totals and the full diff's size, at most 200 paths per section, headed by the full base SHA; once the orchestrator has committed inside the task (each subtask of a decomposed run) a second section lists the change since that commit. A node's request.json carries both under context_paths (change_overview_path, diff_base), and its rendered prompt's context footer lists the overview just before the diff and closes with Change base: <sha> and one where-to-start sentence: start from the overview and inspect files with git diff <base> -- <path> when every provider on the node's route grants a shell, read the full diff otherwise. A fallback attempt handed the partial diff of a failed provider gets none of the three, because they describe a different change.

Use the operator log for live monitoring. Provider stdout.log and stderr.log are finalized and redacted after the subprocess exits, so do not tail them while an attempt is still running.

Troubleshooting node prompts#

  • A role_file edit "did nothing" — confirm you edited the role file the node actually uses. install delivers the built-in flows + their role files under .worc/flows/ (each flow's prompts in its own <task_type>/ subdir), and .worc/flows/ is the only copy the orchestrator reads, so edit the copy there (a custom operator flow likewise keeps its role files under its own .worc/flows/<task_type>/ subdir). Compare rendered-prompt.md against your file.
  • A {placeholder} printed literally — only the allowlisted variables interpolate (see prompt templates); any other {...} is intentionally left verbatim so code/JSON braces survive. A path variable with no value for that node renders empty.

7. Recovery playbook — manual_action_required#

A task ends in manual_action_required (exit 2) when the orchestrator stops safely and needs a human. It is not failed (which is for unrecoverable invalid-task/config/security/git errors). The task file is left in place (not moved to tasks/done or tasks/failed) and automatic continuation is blocked until you resolve it.

Common causes and what to do:

Cause (from stuck.md / logs / cleanup_last_error) Action
Fix budget exhausted — max_fix_cycles (or a flow budgets / inline budget: entry) or the global max_total_fix_iterations hit: a budget of N allowed N fix rounds and the next fail/rework was refused. Read stuck.md: the last failing check / blocking findings and the final diff. Fix manually on the task branch, or refine the task, then re-submit — or, when more rounds would help (e.g. a rate limit turned fixing into a no-op), rerun --continue --reset-fix-budget, which also starts every non-blocking evaluator's max_rework_per_stage count afresh. The global backstop cannot be reset.
A loop cut short by a no-progress guard — limit_exhausted is no_file_change or repeated_findings. no_file_change: read the latest fixing output for the blocker that kept the agent from editing. repeated_findings: read the repeated finding first — a request the node cannot satisfy (an input it may not edit, a judgement the flow cannot meet) never converges however many rounds it gets; fix the task, the role prompt or the environment, then rerun --continue (it keeps counting the same loop).
A tool node repeated the same failure — the park message quotes the repeated output and says no top-level findings array was returned. With neither findings nor data the second byte-identical fail parks; with data alone the flow's declared fix budget for that loop is the limit. Make the tool report typed findings, or fix what it measures, then rerun --continue.
Background work lost (background_work_lost, Claude) — the agent ended its turn while backgrounded subagents it started were still running, and the CLI stopped them at exit; the error names each stopped task. Not fallback-eligible and never a park: an agent node ends the task failed, an evaluator manual_action_required. The attempt's cost is still recorded. Keep agents.providers.claude.background_tasks: false (the default — the CLI is launched with CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1, so subagents run in the foreground and this cannot happen); if you set it true, make sure the role prompt never ends a turn waiting on background work. Then rerun --continue.
Checks gate incomplete — a required toolchain could not launch, or every selected set was skipped (including the skip_if_unavailable set that was the only one the diff selected). Install the toolchain on the host, or narrow/extend the selecting paths, or disable that checks node for the task (nodes.<id>.enabled: false). A fix loop cannot install toolchains, so this never routes to fixing.
Git control state drifted (WRI-009) — the index / HEAD / task ref / repo-local config / hooks changed across a workspace-write attempt. Inspect git status and the repo-local config/hooks; reconcile by hand, then rerun --continue. Note this does not park on a read-only node holding the git-evidence grant — there it only warns and continues.
Terminal cleanup unsafe — base-branch checkout would lose uncommitted work or the branch state is ambiguous (§8.3). Inspect the repo (git status); reconcile by hand, commit/stash or discard intentionally, return to base_branch, then re-run watch.
More than one active task on restart (inconsistent state, §13). Only one task may be active. Decide which to keep, mark the others resolved, then re-run.
Exchange contaminated (WRI-002/007) — the agent mutated the read-only exchange; the tree was quarantined to .worc/runs/exchange-quarantine/<task-id>/. Inspect evidence.json (expected vs observed, plus created_at / node_id / attempt); tasks.quarantine_refs lists every bundle the task produced. Automatic recovery will not resume over it. Once you have read the diagnosis, rerun --continue clears the contamination flag and continues (the evidence is kept), or a fresh rerun discards the contaminated tree and re-derives the exchange.
Terminal exchange cleanup blocked — the snapshot sealed but the active .worc-io/<task-id>/ could not be removed (a Windows lock/read-only). Later launches are blocked (exchange_active_unsafe). Clear the lock / remove the stale .worc-io/<task-id>/ by hand (the verified seal-* snapshot is safe), then re-run.
Provider tree not proven quiescent (WRI-012 containment_unverified) — a background/detached descendant may still be writing, so the terminal seal, commit/push, and base-branch cleanup are all withheld and the tree is left in place (cleanup_last_error reads "not proven quiescent"). Later launches are blocked (exchange_active_unsafe). Find and kill the surviving descendant (the diagnostic records pids), then remove the stale .worc-io/<task-id>/ by hand and re-run. The uncommitted work stays on the task branch working tree.
Exchange not clean at start — a stale/foreign entry in .worc-io/ (e.g. a prior task whose seal was interrupted) blocks the next launch; the new task ends manual_action_required naming the offending entry, without launching any provider. Remove the named .worc-io/<entry>/ by hand (a verified seal-* snapshot under .worc/ is safe to keep), then re-run.
Codex sandbox could not enforce (WRI-003 capability_unavailable) — before each codex exec, a no-model codex sandbox canary proves the generated profile is OS-enforced. On an old Codex CLI or a host missing the sandbox helper it cannot run, so the attempt fails pre-model with capability_unavailable (fallback-eligible only to a provider that can itself isolate the node, else manual_action_required). Upgrade/repair the Codex CLI (on Windows see the sandbox-helper entry under Windows troubleshooting); run worc preflight, whose live capability smoke reproduces this before a task runs. Or route the node to Claude / opt out with strict_isolation: false.
Codex permission profile not enforcing (WRI-003 configuration_error) — the canary found a path that should be denied but was actually readable/writable: the profile is not enforcing. The same class covers Codex refusing to load the generated profile (its own first-line Error: naming a config, profile, permission or filesystem path problem — permission-profile canary FAILED: codex rejected the generated profile before running probe …), at either value of strict_isolation. A non-fallback security result — the run stops, no other provider is tried. A leak is a real isolation failure, not a flake: do not weaken the profile. Check for a conflicting managed/MDM Codex policy or a tampered install; worc preflight's capability smoke reports the leaking probe. A rejected profile is ours to fix — read the quoted Error: line, then correct the config it names or upgrade the Codex CLI; worc preflight's smoke builds the same profile a real attempt launches, so it reproduces the refusal. Fix it, then re-run.

A §19-rejected task is different: it is terminal failed, quarantined to .worc/tasks/rejected/ (under the gitignored home, so a rejected task never rides the audit commit) with a validation_report.json (and a validation_reason in the ledger), and never gets a branch. Fix the task file (e.g. add a Description, a valid id, remove injection-shaped front-matter) and re-submit by promoting it from tasks/preparing/ (or dropping the corrected file back into tasks/pending/). worc list --all lists such ids under rejected: with the reason (see worc list); a re-submission under the same id is accepted, and the id leaves that section once it has a row.

A finished task's own file reappearing in tasks/pending/ is not a failure. With the lifecycle tree tracked and audit_on_branch: task, promote left your commit of the file on base while the terminal move into tasks/done/ was committed on the task branch — so returning to base restores the pending copy. The orchestrator recognises it by content (tasks.source_sha256, newline-normalized so a core.autocrlf checkout still matches): the daemon skips it, worc run on it refuses (exit 1, run: <id> already reached a terminal; its task file was left in place. …, pointing at rerun / finalize), and it is neither re-run nor quarantined. A reject of an id that already reached a terminal status appends no second ledger record and sends no notification. Only a file that merely reuses a settled id — different content — is still quarantined (id already used by a <status> task; give this file a new id), because left in the queue it would be re-rejected every tick.

Recovery is idempotent: re-running watch/run resumes the single in-flight task, reuses the existing branch, continues from its persisted flow checkpoint ({completed_nodes, current_node, loop_counters, publish_operations} — not a per-stage status, those are gone), and never re-commits/re-pushes a completed operation. A resumed fixing/review node is handed the same inputs a fresh run produced — they are rebuilt from disk plus state.db (current.diff and its change-overview.txt — the change base is read back from the overview the run wrote, not resolved again from a clone that may have moved — plan.md, the newest quality-failed checks/<NNN>.log, and the findings of the last recorded in_flow_verdict) and re-published into the current task's exchange, so the agent reads exchange paths exactly as it would on a first pass, and the fix counters are not incremented a second time.

The tasks/ lifecycle tree at the repo root is where live task files belong; it is gitignored by default and becomes the git-tracked audit trail only if you asked install to track it (see §5). The runtime home .worc/ and the agent-facing exchange .worc-io/ are always gitignored by install (all three lines appended to .gitignore), so their contents never ride a commit.