Operations: Running Tasks#

Part of the operations guide: run and watch, what a provider outage does, the commands that act on a task in flight, and how to stop the daemon safely.

4. Running#

python -m wastech_orchestrator run tasks/pending/task-001.md   # one task, end to end
python -m wastech_orchestrator watch                           # resume + process pending tasks
python -m wastech_orchestrator --log-level debug run PATH/TO/task.md  # more verbose operator trace
python -m wastech_orchestrator status                          # active/latest persisted task

run takes a path to a task file (.md/.json), not a task id — run p7-04-compile-tool fails with No such file or directory. This is the one command keyed by path: it starts a brand-new task from the file, so no id exists to look it up by yet. Everything that acts on an already-registered task — rerun, status, finalize — takes the task id instead.

Two refusals keep run from quietly misusing a file. Pointed at a file that is a terminal task's own leftover (recognised by content, the same guard watch applies), it exits 1 with run: <id> already reached a terminal; its task file was left in place. Use `worc rerun` to re-attempt it, or `worc finalize` to close it out. — rather than rejecting it as a duplicate id and quarantining your own file. And a gate reject that does move a file says where it went: a moved to <path> line under the rejected — line, plus a task file quarantined WARNING naming the destination, because the quarantine is under the private home you do not browse.

run and rerun are visible executors. For its whole duration either one writes .worc/orchestrator.run, so status, list and top show its task as running rather than parked (no daemon), and every "is anything executing here?" guard answers yes. (Before, only run wrote it, so a task being driven by rerun read as parked (no daemon) — the label external monitoring takes for a finished run — while its node was still working.) That marker is deliberately not the daemon's own orchestrator.pid: stop reads that file and asks the recorded process to finish, while neither command installs stop wiring — no SIGTERM handler, no stop-file poll — so a stop aimed at it would wait out its timeout and then escalate to a kill on a process that was working correctly. Two files let every liveness probe answer while the stop ladder keeps addressing only the process that can answer it. The marker is reaped in a finally.

The commands that drive the pipeline or git in the shared clone — watch, run, rerun, finalize, merge-task, prs --sync — refuse while either executor holds the home, and the refusal names the owner and the advice that fits it: the daemon is stoppable (worc stop), a run or rerun is not, so the only honest instruction is to wait for it or interrupt it where it runs (a 'run' or 'rerun' is executing a task in this clone (pid <N>); wait for it to finish, or interrupt it where it runs). A second run or rerun in another terminal is refused too, and that refusal does not depend on timing: each executor claims its marker with an exclusive create (O_CREAT | O_EXCL), so of two runs/reruns — or two watch daemons — racing for the same home the kernel picks one, and the loser refuses (exit 1) naming the holder's PID without touching the winner's marker, stop sentinel or agent. The owner check in front stays for the better message in the common case. A marker whose holder is dead, or alive with a different start-time token (a recycled PID), is reclaimed, so one crash never blocks the clone for good — except on Windows, where liveness cannot be probed and a marker's presence is the whole signal: a marker a hard kill left behind there refuses until you delete it.

A read-only plan is never refused for a busy clone. merge-task --dry-run, finalize --dry-run and rerun --dry-run mutate nothing, so they print while an executor owns the home — which is exactly when an operator wants to read them, since with auto_mode.enabled: true and git.auto_merge: false the daemon is up by definition. The exemption is not silent: each dry run prints the owner as a note beside the plan, because past the guard plan_rerun's premise ("no executor owns the slot") no longer holds and a confidently wrong plan would be worse than a refusal. The real operations still refuse.

Staging a task (promote). watch never scans tasks/preparing/, so a task file can be composed there without being picked up mid-write. worc promote <id|file> is the explicit "it's ready" step: it atomically renames the staged file into tasks/pending/ (a decomposition root pulls the subtask specs it references along with it), and worc promote --all moves every staged top-level file plus the whole subtasks/ subfolder. The same verb exists inside worc shell. (worc clear is the unrelated screen-clearing convenience — it deletes nothing.)

watch respects orchestrator.auto_mode.enabled: off (default) it processes/resumes one task and returns the working copy to repo.base_branch (unless the branch mode / repo.checkout_base_on_cleanup keeps it on the working branch); on, it processes pending tasks sequentially, running terminal cleanup between them. A manual_action_required outcome always blocks automatic continuation. A separate case is a terminal task whose interrupted cleanup cannot complete on resume (e.g. a dirty foreign tree, a failed base checkout, or a non-quiescent tree): that task is already terminal and holds no slot, so it does not freeze the queue (VF-21) — watch records the block (cleanup_last_error), logs a warning naming the held task, and continues scanning pending tasks; the cleanup is retried each tick and clears on its own once the working tree is resolved (rerun/finalize). Exit code: 0 done, 1 failed, 2 manual_action_required, 3 paused (a task soft-parked on a provider outage — see provider outage behavior).

auto_mode governs task sequencing, not human-in-the-loop. It does not relax HITL: a node that escalates — an embedded planning/refinement question/approval, or a dangerous-diff approval — still blocks the run until a human answers, or until telegram.ask_timeout_s elapses (default 28800 s / 8 h), after which the wait resolves to manual_action_required. So an unattended watch completes only when the pending tasks are well-specified enough that no node needs to ask (see task authoring → planning escalation).

Unattended night runs (the worc-night-run skill)#

watch chains tasks only while nothing goes wrong: a failed, a manual_action_required, a missing toolchain or a dirty index stops the queue until a human looks. Clearing those is judgement work, so the third unattended shape is not a command: install (and upgrade-docs) deliver a copy-ready agent skill, .worc/guide/skills/worc-night-run/SKILL.md, for an agent that supervises worc itself. Claude Code discovers skills only under <repo>/.claude/skills/, so copy the folder there once (cp -R .worc/guide/skills/worc-night-run .claude/skills/; PowerShell: Copy-Item -Recurse -Force .worc\guide\skills\worc-night-run .claude\skills\) and again after an upgrade-docs brings a newer version.

What the skill does, in outline — the file is the contract:

• Before the night, it runs the gate once and prints a night plan: worc preflight and worc validate-flow --all must exit 0, no executor may own the clone, nothing of yours may be staged, the queue order is read from worc list --pending --format json (rank), and a pending file whose id already has a terminal row is resumed with rerun, not started. It names the settings that fail closed without a human (branch_mode: current, a hitl node with Telegram off, confirm_next_task, trust_level: strict, auto_merge: true) and leaves out tasks whose depends_on cannot merge tonight.
• The loop runs one task at a time with worc run <tasks_dir>/pending/<file> in the background, polls worc status <id>, and reads the verdict off the exit code (0 done, 3 parked — waits and resumes, spending no attempt — 1 failed, 2 manual or a config error).
• Each task gets four starts on an escalation ladder: start; restart only (rerun --continue, optionally --reset-fix-budget / --from); repair the environment, never the work; and, last and on the record, a mechanical edit of an operator-owned input (a task file's front-matter shape, one role prompt, a non-security config key). A security refusal (containment_unverified, capability_unavailable, a new exchange quarantine, a changed push destination, injection_suspected, …) abandons the task at once; a host-wide failure stops the session.
• Hard limits: it never commits, pushes, opens or merges a pull request, never weakens the security envelope, never deletes quarantine evidence or a failed task's artifacts, never runs two tasks, and never edits a task's deliverable to make checks pass.

It leaves one directory per session, .worc/night-runs/<YYYYMMDD-HHMMSS>/: a <task-id>.md write-up per task, written the moment the task reaches a terminal, a morning report.md, and its own state.json so a re-invoked session continues the night. The orchestrator never writes that directory and no clean command prunes it.

Provider outage behavior#

When a coding-agent CLI dies mid-task with a transient server error (a 5xx / network blip classified provider_unavailable or network_unavailable), the orchestrator no longer throws the task away. It recovers in three bounded, audited steps, all tuned by agents.retry:

1. Same-provider retry — retry the same provider with exponential backoff (base_delay_s, doubling, capped at max_delay_s), up to max_attempts times. The CLIs already retry 5xx internally, so the value here is the longer wait window, not a tight loop.
2. Symmetric fallback — if retries are exhausted, switch to the other allowed provider (Claude↔Codex) and retry it the same way. (With a single allowed provider there is no switch target.) A node that finishes on the fallback provider is never silent: the run log carries a WARNING (node ran on a fallback provider — routed to <a>, ran on <b> after <a> failed (<error_class>); its result stands, but it is not the run the flow planned, with — the flow pinned this node's provider before the semicolon when the flow declared the provider, in which case the router's own falling back record is a WARNING too rather than INFO), telegram.trace sends a ⚠️ ran on a fallback provider: … trace, and the PR body gains a ## Provider fallbacks section (see the PR body). The result stands; it just is not the run the flow planned — a critic pinned to a second model that ran on the first one did not perform the cross-check.
3. Soft pause — if every provider is unavailable, the task is parked as resumable rather than failed: it stays active, watch shows it as running (paused), the checkpoint is preserved, and the next tick / restart resumes it from where it stopped (no work or commits are lost). A one-off worc run exits 3 and leaves the task resumable for the next run/watch/restart. The pause is bounded by max_blocked_s (default 6 h) — once exceeded, the task goes terminal failed so nothing hangs forever. When the exhausted provider named the instant its own window reopens (a rate-limit reset banner), that instant is recorded in tasks.blocked_until and the intervening ticks are cheap no-ops instead of full re-entries. It is untrusted input: absent, unparseable, or already-past values fall back to retry-on-every-tick, and a claimed window is clamped to the max_blocked_s ceiling — so it can only ever shorten a wait, never extend one. An operator stop is excluded (a cancelled run resumes when the operator says so).

A subscription/session rate_limited (HTTP 429 / a "session limit … resets" banner, on either provider) follows the same recovery: it is not tight-retried on the same provider (a rate limit wants a long defer, not a hot loop), but it is fallback-eligible — the other provider draws on a separate quota and usually finishes the work — and if every provider is rate-limited it parks (step 3) and waits out the reset window, rather than failing or marching the queue into the wall. The 6 h max_blocked_s default comfortably outlasts a provider's ~5 h usage window; because the parked task holds the single slot, watch also stops picking up new tasks until it clears (the queue self-throttles). timeout is still not treated as transient (it may have done long/partial work) and takes the existing fallback/terminal paths. Quality failures are never retried. Every attempt is in the provider_attempts audit trail.

Listing tasks and shell completion (list / completion)#

worc list is a read-only snapshot of what exists, so you do not have to recall an exact task_id to act on it. With no flags it prints three sections — the active task, the tasks/pending queue, and the most recent terminal tasks. Focus it with --pending, --recent [N], or --all (every known task across all statuses, plus a fourth section, rejected: the ids the validation gate refused). It opens state.db read-only — safe to run while a watch daemon is live — and writes nothing; the ledger behind the rejected section is read the same way.

The pending section is the queue in the exact order watch will run it: filtered to this instance's orchestrator.queue, ranked high → mid → low, and broken by natural (numeric-aware) filename order within a priority — each row shows its rank position and the priority/queue it sorted on (a file with no parseable id is shown by filename). This is the same sequence worc top and the console ps view display. Read the run order from worc list / worc top, not from your file manager or IDE — a file listing shows bytewise filename order, which disagrees with the scheduler (it puts p10-… before p9-…) and even differs between operating systems, and it knows nothing about priority, queue, or depends_on.

worc list                       # active + pending queue + recent terminal tasks
worc list --recent 20           # only the last 20 terminal tasks
worc list --format json         # structured output for scripting (one JSON array; see below)
worc list --all                 # every known task + the `rejected:` section (reason in parentheses)
worc list --format ids --scope rerun   # bare ids a rerun would accept (failed / manual_action_required)

--format ids prints one bare task_id per line (notices go to stderr) and --scope {rerun,status,finalize} narrows it to the ids that command accepts — the machine-readable surface the completion script consumes. Combined with a section flag (--pending/--recent/--all) it lists the ids of exactly that section, from the same disk-plus-DB source as the table view — so worc list --pending --format ids prints the queued task ids (including freshly-queued files that have no state.db row yet), not just the DB-derived set. A bare --format ids (no section, no scope) stays DB-derived.

--format json is the surface meant for scripts — a tracker connector, a dashboard, a cron job — and its entry shape is the contract. Every entry carries task_id, status, title, branch, and pr_url — the URL of the pull request the task opened, or null (a file-derived pending entry has no row yet, so its title, branch, and pr_url are always null, and it adds file, rank, priority, queue). Two things a consumer must know. First, status is a display label, not the raw status: running (paused) / running (paused until <timestamp>) during a provider-outage park, parked (no daemon) for a running row whose executor is gone, and done (recovered: <loop>) for a finished task whose fix loop a guard had to cut short (see §6 → why a task is stuck) — match the leading token. Second, --all lists database rows only, and its rejected section is derived read-only from the ledger: an id whose only ledger trace is a validation reject and that has no tasks row appears once, built from its newest record — {task_id, status: "rejected", title: null, branch: null, pr_url: null, validation_reason, rejected_at}. No ledger file means an empty section (exit 0); an id leaves the section the moment a corrected file is accepted under the same id, because from then on it is an ordinary task with an ordinary row. In the table view those ids print under a rejected: heading with the reason in parentheses, and when any listed task has provider attempts with no recorded cost the table ends with one footer line — note: Codex cost not accounted (13 attempts, 8.4 M input tokens) — that the json/ids views do not carry (worc status prints the same fact as cost_not_accounted=…). The default view, --recent, --pending, and --format ids never carry rejected ids — no verb accepts one. Check the exit code before parsing: an unreadable config, an incompatible state.db, or (under --all) a ledger carrying a torn or non-JSON line exits 2 with a plain error: … naming the file (LedgerUnreadableError) where the JSON would have been.

worc completion bash|zsh prints a completion script you wire once. It completes subcommand names and flags statically, completes run with task files, and completes the id of status / rerun / finalize dynamically by shelling out to worc list --format ids --scope <command> — so the id list is always live and there is no extra dependency to install.

# zsh
source <(worc completion zsh)            # or: worc completion zsh > ~/.zsh/completions/_worc
# bash
source <(worc completion bash)           # or drop into ~/.bash_completion.d/

Live monitor and interactive console (top / shell)#

To watch a run without juggling terminals or re-running status, worc top is a single auto-refreshing, read-only surface: the active task + flow node, a parked marker (a task soft-paused on a provider outage) or gate pending marker (a durable max-turns gate awaiting the operator), the pending queue (filtered to the served queue and priority-sorted, exactly the order the daemon will run), recent terminal tasks, and a tail of the daemon --log-file. It polls state.db read-only — safe while a daemon is live — and quits on q (then Enter). It is a client over the daemon; it never starts the engine. Stdlib-only (no extra):

worc --log-file ./logs/daemon.jsonl watch &        # the daemon writes its log here
worc top --log-file ./logs/daemon.jsonl            # match --queue to the daemon's queue if you set one

--log-file is global: on watch it precedes the subcommand (worc watch --log-file … is an argparse error). On top/shell it is a subcommand flag naming the file to tail.

worc shell is the interactive console layered on the same view (it needs the optional [shell] extra — pip install wastech-orchestrator[shell]; the daemon and headless installs never import it). Entry is passive: it attaches to a live watch daemon if one is running (left running on exit), otherwise it opens idle with the queue not being served and says so — it never auto-spawns and never claims a task on entry. The operator starts serving explicitly with up (alias watch), which spawns a daemon and verifies it actually came up — it polls for the daemon's PID file (its liveness signal) and, if the daemon dies at startup, prints the real error captured from its startup log (.worc/logs/daemon-startup.log) instead of a false "started". The console streams the daemon --log-file above a prompt and dispatches commands onto the existing verbs: enqueue <file> (atomically copies the file into tasks/pending via a temp file + os.replace, so the watcher never sees a half-written task; queued, not run until up — input is never blocked and paths keep spaces/backslashes intact, so Windows absolute paths work), promote <id|file> / promote --all (atomically moves a staged task file from tasks/preparing/ into tasks/pending/ — a decomposition root pulls its subtask specs, and --all moves every staged top-level file plus the whole subtasks/ subfolder), ps (the live view, alias jobs), status/logs/tasks/prs/merge-task/finalize/rerun, up/watch/down/restart, cancel <id> (de-queues a pending file to .worc/tasks/rejected/; an active task has no per-task cancel), clear (wipe the console screen and scrollback — no files touched), and quit. Slot-guarded verbs still apply: merge-task/finalize/rerun refuse while the daemon is up, so stop it (down) first. quit detaches — the daemon and any in-flight task keep running (a long task survives closing the panel), and reopening worc shell reattaches; when the daemon is still serving the queue it first prints a loud warning and, in an interactive session, asks for confirmation before exiting (scripted/headless mode prints the warning but never blocks). To stop the daemon use down (soft) / down --force-full (hard) or worc stop from another shell. The default daemon log is .worc/logs/daemon.log (or --log-file), so attach/up can always tail without hand-coordinating a path.

Stopping the daemon safely (the stop ladder)#

worc stop and worc restart (and the console's down) follow a stop ladder keyed on whether a task is active:

• Idle (no active task) → stops immediately, no prompt — any form.
• Busy, no flag → refused. An interactive terminal prompts for a literal YES (maps to the soft stop) and names --force-full as the way to interrupt the running agent now; a non-interactive caller (CI/pipe, or --non-interactive) exits non-zero and must pass a flag. The console forwards its down/restart with --non-interactive so a busy stop never drops into a blocking YES prompt that would fight the REPL's stdin.
• Busy, --force (or typed YES) → soft stop: the current flow node (including its post-node bookkeeping) finishes, the engine checkpoints the untouched next node, parks the task resumably, and the daemon exits. The grace period does not interrupt a running agent; to stop now, use --force-full.
• Busy, --force-full → hard stop, on both platforms: the active agent's whole subtree is reaped immediately along with the daemon, and the task resumes from its checkpoint on the next start. On POSIX the daemon records each agent's (pid, pgid) at launch (in .worc/orchestrator.children); a hard stop SIGKILLs the daemon's own process group and the recorded agent's group, plus a recursive descendant sweep as a backstop for any child that broke away into its own session/group. On Windows it taskkill /F /Ts the daemon's process tree (and the recorded agent's) — the agent is a child, so the tree-kill reaches it; the console that launched the daemon is its parent and is untouched.

--timeout SECONDS (default 30) is the cooperative grace period. A soft stop never hard-kills on either platform: on POSIX it sends SIGTERM for an immediate wakeup, and on Windows the request rides the orchestrator.stop sentinel because os.kill cannot reach an unrelated watcher. The daemon polls that sentinel during idle sleep and the FlowEngine checks the same cancellation predicate before every node, so a normal soft stop no longer waits for the entire task graph. If the grace period still expires inside one long node, the stop stays pending rather than escalating: the daemon still exits at its next node boundary, --force-full remains the only immediate interrupt, and the PID/stop/children handles are all retained so a concurrent or later --force-full never loses its target and restart cannot start a duplicate watcher. A legacy PID-less state with either retained stop or children handle is ambiguous too: restart fails closed instead of clearing the sentinel by starting another watcher. A stop-killed agent is classified as cancellation (not a crash), so the router never falls back to a fresh agent and the task is parked resumably rather than reported failed.

No agent is ever orphaned. A soft --force never kills — even on timeout it stays a pending graceful stop and lets the daemon reap its own agent subtree when it exits at the next node boundary. Every hard route reaps the subtree immediately via the recorded agent handle plus one subtree-kill primitive: --force-full, a run_process timeout, and the daemon's own exit (a foreground Ctrl-C or a crash reaps the agent subtree in the daemon's finally before the PID file is removed). The agent now leads its own session/group (reversing the earlier shared-group topology), so a foreground Ctrl-C no longer relies on group inheritance to reach it — the daemon reaps it deterministically instead. POSIX is the validated acceptance gate; a live Windows smoke has confirmed the timeout path against a real temporary parent/child process tree and its runtime-file cleanup. The owner-level end-to-end smoke with a real watcher and coding-agent CLI remains pending.

Every provider attempt ends with a proven-quiescent process tree (WRI-012). Beyond the stop routes above, run_process now runs a quiescence barrier on every exit path — clean exit, non-zero, timeout, interrupt — not only on timeout/interrupt: it terminates the attempt's process containment and proves it empty within a bounded budget before the result is trusted, so a background/detached descendant that outlived the CLI cannot keep writing the repo after the attempt returns. On POSIX the containment is the agent's session/group plus during-run descendant tracking (a setsid child that leaves the group is tracked while still parent-linked and killed by pid); on Windows it is a kill-on-close Job Object that still owns descendants after the root exits (where taskkill /T alone cannot prove the tree gone). If the tree cannot be proven empty (a stuck/unkillable descendant), the task ends manual_action_required — never a quality failure and never a fallback — with a secret-free diagnostic (platform, a member count, and pids), and the recorded agent handle is deliberately kept so a later --force-full/recovery can still reap the survivor. Crash-semantics note: the Windows Job Object is kill-on-close, so if the orchestrator process itself dies mid-attempt the in-flight agent tree dies with it (recovery then treats the task as parked/crashed exactly as before); POSIX crash behaviour is unchanged (the agent survives a daemon crash and is reaped via the children-file on the next stop/recovery).

The control plane is frozen per task (WRI-010). At task start the orchestrator snapshots the exact control inputs the task's flow uses — the flow YAML, every role/supervisor prompt, and each tool node's executable — into a private immutable bundle under <private_home>/runs/control-bundles/<task-id>/, and every later node reads the frozen copy, not live .worc. What this means operationally: (1) editing a flow, role prompt, or tool in .worc/ while a task is running does not affect that task — the change is picked up by the next fresh task (or a rerun), so operator authoring on the live control plane is unchanged; (2) an operator rerun --continue adopts the current on-disk control plane — it re-freezes the flow/role/tool bundle from live and resumes, so a flow/role/tool fix you edited between runs takes effect from the resume point onward (this is the flow-tuning loop); only automatic crash-recovery keeps the task's original frozen bundle and reports a parked live edit as manual_action_required (a crash can follow an agent's mutation, so it is never silently adopted); (3) if an agent (or anything else) modifies a control file mid-run, it is detected after the attempt and the task ends manual_action_required — never a fallback. Install/upgrade (install, upgrade-docs) and interactive operator editing all keep using the live .worc/ control plane as before.

The agent's task input is frozen per task; repository instructions are read live and freely editable (WRI-011, amended by VF-5 and VF-20). At task start the orchestrator snapshots the agent inputs whose identity must stay stable — the validated task packet and the root repository instruction files — into a private immutable bundle under <private_home>/runs/instruction-bundles/<task-id>/ under one instruction_manifest_digest; the agent reads only redacted exchange copies of those. Repository instructions (AGENTS.md, AGENTS.override.md, CLAUDE.md) are handled differently after VF-5: they are not injected and provider-native discovery is not disabled — each agent reads the repo's live root files itself (Codex via native AGENTS.md discovery; Claude via its Read tool, since its --setting-sources "" lockdown also turns off native CLAUDE.md auto-load). These files are ordinary, editable repository content (VF-20): a workspace-write attempt may change AGENTS.md/CLAUDE.md or anything under .agents/rules/ — the change lands as a normal proposed diff, never blocked. A per-source freeze of these files at task start still folds into the instruction_manifest_digest for audit (the record of what the agent read), and when a run's diff touches a governance path the orchestrator loudly notifies the operator — a WARNING in the run log/console, a ## Governance files changed section in the PR/commit summary, a governance_changed field on the completed-ledger record, and a governance= mark on the Telegram completion message — so a run that edits its own rules is always visible, never silent. What this means operationally: (1) editing a task file while a task runs does not change that task's frozen input — a later node/evaluator/supervisor/resume/fallback still sees the snapshot; the edit is a normal proposed diff adopted by the next fresh task. It is no longer silent: every resume compares the live file against the frozen packet and, when they differ, logs a WARNING naming the instant the packet was frozen (frozen_at in the bundle manifest, kept outside the digest) and stating that worc stop plus a fresh start applies the edit, and the same sentence lands in summary.md under ## Task file changed after the run started; with the lifecycle tree tracked, expect such a run to end manual_action_required, because the audit commit refuses a lifecycle file that no longer matches the packet; (2) editing AGENTS.md/CLAUDE.md between runs is picked up automatically on the next run (the agent reads them live) — there is no instruction freeze to re-adopt; a task may also edit them (and .agents/rules/**) during a run as ordinary work — the orchestrator reports the change to the operator rather than blocking it (VF-20), and an operator who wants an always-ask step on these paths can add them to security.protected_paths; (3) a task whose required input contains a known secret value (a resolved env / denied / memory secret) fails before the model runs with an actionable error. Tasks whose deliverable is editing repository guidance or rules work end to end — the edit lands as an ordinary proposed diff, the operator is notified of any governance change, and the live files remain readable repo source.

Git control state is orchestrator-owned per task (WRI-009). Every orchestrator git command runs with a private empty core.hooksPath and editor/pager/signing/fsmonitor/filter neutralization, so a target-repository hook or an agent-set .gitattributes/config filter can never execute inside an orchestrator Git subprocess (an untrusted repo-local clean/smudge/filter/diff driver — or core.sshCommand/credential.helper — stops the run in manual_action_required; operator global git config, e.g. a global git lfs install, stays trusted). Before every commit the complete staged set is proven to contain only that operation's files (a bare git commit commits the whole index, so the scoped git add is never trusted alone), and the audit commit refuses a tasks/<id>.md whose content no longer matches the task the run was authorized from. What this means operationally: (1) .worc//.worc-io/ and another task's lifecycle files can never ride a commit, even force-staged; (2) an operator who pre-stages files in the shared clone before a run is asked to unstage/commit them first (the run refuses rather than sweeping them in — unstaged edits are left untouched); (3) git ≥ 2.9 is required under strict isolation (older git ignores core.hooksPath), enforced at run/watch/rerun startup. The provider-side read-only-gitdir grant / write-deny projection lands with the Codex/Claude isolation tasks.

Drift across an attempt is reported, and never parks the task — on any node class. Around every attempt that gets a shell the orchestrator fingerprints Git control state (index, HEAD, the task ref, repo-local config, hooks, merge/rebase/bisect markers) and re-checks it once the provider tree is proven quiescent. A change is a loud WARNING carrying the redacted aspect summary plus a ⚠️ Telegram trace — done (node changed git control state), or done (node wrote to the workspace unexpectedly) for the working tree — and then the run keeps going. The earlier model, in which drift ended the task manual_action_required with one bounded exception for a read-only node holding the git-evidence grant, is gone. Why, and what the fingerprint reads beyond the clone's own .git, is in worc_architecture.md §8.

What that asks of you: the drift warning is a stop-the-run signal, not a note. If you did not do it yourself, stop the run and discard the clone before anything is committed or pushed — the orchestrator's own next git command executes in that clone. What holds a commit made inside a run is a different mechanism: the pre-publish dangerous-diff gate, which measures from the last commit the orchestrator itself made and therefore sees it.

Re-attempting a terminal task (rerun)#

A task that ended failed or manual_action_required is terminal — watch/resume never pick it up again. A task killed mid-flight (stopped, or its daemon died) is left at DB status running; rerun now accepts that stale running row directly too — no finalize --as failed dance first — because the daemon must already be stopped to run rerun (so the running row is provably not live). rerun re-attempts without hand-editing state.db, the ledger, or git. It needs an idle slot (no other active task) and no live executor — neither the watch daemon nor a worc run or another rerun — since it drives the pipeline in the shared clone, and while it runs it holds the same orchestrator.run marker as run (see §4), so status/list/top report its task as running; rerun --dry-run is exempt and prints the plan with a note naming the owner; it records a new ledger entry linked to the prior attempt (attempt, rerun_of). attempt is counted from the ledger when the record is appended — the record's ordinal among that id's terminals, gate refusals excluded — so an interrupted run that was resumed appends once, and a rerun's number is whatever terminal it turns out to be. From worc shell the console forwards rerun with --non-interactive too (the same reasoning as the stop ladder above): pass --yes on the command line, since an unanswered confirmation prompt would otherwise fight the REPL's own stdin reader instead of asking [y/N].

worc rerun task-001 --dry-run        # show the planned reconciliation; write nothing
worc rerun task-001 --yes            # fresh attempt from the current base_branch
worc rerun task-001 --continue --yes # fix-and-continue: reuse the branch, re-enter at the failed stage
worc rerun task-001 --continue --reset-fix-budget --yes   # + grant a fresh fix budget (exhausted fix loop)
worc rerun task-001 --continue --from implementation --yes # + re-enter at a chosen node

• Fresh (default, new-mode branch) — for a quality failure, a clean redo, or when base_branch has moved on: the agent branch is reset to the current base (the stale local branch is deleted and recreated), the per-attempt state is cleared, and the prior attempt's logs/<id>/ is archived to logs/<id>/attempt-<N>/. The prior attempt's node runs are deleted with it, so the rework rounds a non-blocking evaluator spent there no longer count against its max_rework_per_stage in the new attempt. If a prior remote branch or open PR exists it refuses and points you at the finalize command; pass --force-reset-remote to delete that remote branch (which closes its PR) instead. A fresh reset only happens on a new-mode branch (the one the orchestrator owns).
• Restart in place (a plain rerun, no flag, on an operator-owned branch) — when a task in branch_mode: existing/current failed before any node ran (no flow checkpoint — e.g. a transient missing flow / provider outage at pickup), a fresh reset is forbidden (the branch is yours), so rerun instead re-drives the whole flow from the top on that branch as-is: it clears the per-attempt state and archives the prior logs/<id>/, but never touches git, so commits already on the branch are preserved. rerun selects this automatically (the confirmation reads [restart] on branch '<branch>'); the retry is still ledger-linked as the next attempt. If the operator-owned run did produce work (a checkpoint exists), a fresh rerun refuses and directs you to --continue instead.
• --continue — for an infrastructure failure you fixed by hand (a missing tool, PATH, a dropped Telegram approval) or to apply a flow/role/tool fix you edited between runs: it keeps the existing branch and the work already done, adopts the current on-disk control plane (re-freezes it, so your edits take effect from here on), and re-enters the pipeline at the stage it failed, reusing the same resume engine as crash recovery. Only available when the failed run recorded a recoverable stage. Once a node that writes to the tree has run — an agent node, or any evaluator / checks / publish node downstream of one — --continue tolerates the task's own uncommitted working tree, because that work is the legitimate input to those nodes, and notes that the WIP is preserved and committed into the task once the flow reaches publish. That is deliberately wider than "the task reached review/testing/fixing/publish": a run parked mid-writing, before its first critic ever ran, resumes on the work it left on disk instead of being asked to throw it away. Before any writing node has run — at planning/refinement — a dirty tree is still refused as unaccounted work, as is any fresh rerun. If a resumed node re-parks to manual_action_required before publish, the task's own WIP is preserved on the branch and the real node reason is reported (not a spurious "unaccounted changes"). When the failure was inside the publish node itself (commit_code failed, leaving code uncommitted), --continue re-enters publish and idempotently docommits it, then pushes and opens the PR. A publish git failure logs its git stderr (ERROR, daemon log) and writes a publish-error artifact, so the cause is visible rather than a bare publish_failed.
• --reset-fix-budget / --no-reset-fix-budget (with --continue) — for a task that ran out of fix cycles (manual_action_required, max_fix_cycles), e.g. because a transient rate-limit turned fixing into a no-op: grant a fresh fix budget. It resets the consecutive fix-loop counters so fixing runs again, and every non-blocking evaluator's max_rework_per_stage count: it appends a rework_budget_reset marker row to evaluations (append-only — the earlier verdicts stay in the audit trail), and only rework verdicts after the task's latest marker count against that cap. The dry-run's state: line says so (terminal markers cleared; consecutive fix budget and every non-blocking evaluator's max_rework_per_stage count reset (global backstop kept); subtasks/publish-ops kept). Without the flag a non-blocking evaluator whose count is already spent accepts on its next pass without another fix round — a full, paid evaluator run that changes nothing — so rerun --continue (the dry-run plan and the real run alike) prints a note: naming each such gate on the resume path as <node> (<used>/<cap>). The global max_total_fix_iterations backstop is left intact, so the loop is still guaranteed to terminate — this never grants unbounded retries. If the resume would re-enter a node whose relevant fix-loop budget is already exhausted and neither flag was passed, rerun prompts interactively ([y/N]) before proceeding — this prompt is not skipped by -y/--yes (a budget reset is exactly the kind of consequential decision --yes doesn't cover); --no-reset-fix-budget declines it non-interactively and refuses the resume instead. A task stuck on the global backstop itself is not unblocked by a reset — the dry-run and the real resume both report a hard error (the backstop is a hard ceiling) rather than resetting a budget that would not actually help.
• --from <node> (with --continue) — re-enter at a chosen node of the flow instead of the recorded stage (e.g. --from implementation to re-implement). The node must exist in the flow currently on disk (a plain --continue likewise validates that its checkpoint node still exists, and refuses with an actionable message if your edit removed or renamed it). Because --continue adopts the current on-disk control plane, a note: is printed when it changed ("the control plane changed since the checkpoint; --continue will adopt the current on-disk flow/roles/tools and resume at '<node>'"), and the checkpoint's fingerprint is rebaselined so the resume lands at the chosen node instead of restarting from the top. Jumping back is always safe; jumping forward relies on the inputs that node reads already being on disk.

Finalize a task you handled by hand (finalize)#

When you resolve a failed/manual_action_required task out-of-band — merged the PR yourself, fixed it locally, or decided to drop it — finalize reconciles the orchestrator's bookkeeping to match. It only records and tidies: it never runs the pipeline and never commits/pushes/PRs. Like rerun it needs no live executor — the watch daemon or a worc run/rerun — because it checks base_branch out in the shared clone (--dry-run is exempt and names the owner).

worc finalize task-001 --as failed --note "superseded by task-014"
worc finalize task-001 --as done --pr-url https://github.com/o/r/pull/42   # records the human merge
worc finalize task-001 --as abandoned --note "obsolete"                    # deliberately dropped

It sets the declared terminal status, runs terminal cleanup (back to base_branch when the branch mode / repo.checkout_base_on_cleanup calls for it, fail-closed on an unaccounted-dirty tree — it reports, never discards; otherwise it stays on the working branch), reconciles the operator-facing fix-loop counters from the last flow checkpoint (a task killed mid-flow never reaches the clean terminal that normally mirrors them, so without this status/the ledger would under-report the real review/fix churn), reconciles any node_runs row left running by a hard --force-full kill — closing it to aborted and recording a provider_attempts row for the killed attempt (usage_delta_status='unknown'; no guessed cost) so the aborted run is not free in the audit, logged as a WARNING naming the reconciled nodes — moves the task file to the matching folder, closes any waiting HITL prompt, and appends a manual ledger record. For --as done the PR URL is taken from --pr-url, else the URL a crashed run already recorded; with neither it still finalizes but warns and asks for confirmation. It also runs a read-only gh pr view merge check by default (--no-verify-pr to skip; warns+asks if the PR isn't merged). --as abandoned is recorded as manual_action_required with an outcome: abandoned ledger marker (distinct from a plain failure). The agent branch is kept unless --delete-branch. --dry-run previews the reconciliation; re-finalizing an already-finalized task is refused.

By default watch is a long-running loop (orchestrator.poll_interval_seconds: 300, overridable with --poll-seconds N): each tick it runs git fetch + pull --ff-only on base_branch then re-scans, so a task committed and pushed to git after watch started is picked up without a manual pull (discovery is not limited to the local filesystem). Stop it with Ctrl-C. Set poll_interval_seconds: 0 (or --poll-seconds 0) for a single pass — e.g. when an external scheduler re-invokes watch.

Merging a reviewed PR (prs / merge-task)#

With git.auto_merge off (the default), an orchestrator PR is left open for you to review on GitHub. Once you approve it, merge-task is the human-in-the-loop go-ahead that finishes the job — the counterpart to auto_merge for a PR you reviewed.

worc prs                              # open, un-merged orchestrator PRs awaiting merge (DB-only)
worc prs --check                      # ... enriched with live GitHub state per PR (needs gh)
worc prs --sync                       # reconcile PRs merged externally on GitHub (dry-run)
worc prs --sync --yes                 # ... actually record those merges
worc merge-task task-001 --dry-run    # show the plan (PR, conflicts?); merge nothing
worc merge-task task-001              # go-ahead: update branch w/ base, resolve conflicts, merge
worc tasks --status done              # every known task with status + branch (read-only)

merge-task pulls base_branch into the task branch, then: a clean base-merge is mechanical (push + gh pr merge); a conflicting one runs the operator-editable merge flow (git.merge_flow, default merge, seeded at .worc/flows/merge.yaml) — a coding agent resolves the conflicts and the checks re-run — after which the orchestrator commits the merge and merges the PR. Before the flow runs the orchestrator writes a conflict inventory (logs/<task-id>/merge/conflicts.md, with a redacted exchange copy the role prompts read as {conflicts_path}): every conflicted path with its kind, which sides exist, what the working tree holds, and what would count as a decision — because conflict markers are only one shape of conflict, and a modify/delete, an add/add or a binary conflict leaves one side in the tree with no marker at all. After the flow, every conflicted path must show a decision before anything is staged: a path whose bytes are unchanged from what the merge left, one with nothing readable there, or a submodule pointer (a commit id no file edit can express) makes merge-task abort the merge, commit nothing, leave the PR open, and exit 2 with merge-task: manual action required — refusing to commit the base merge: N of M conflicted path(s) carry no decision. and the paths. That refuses a correct "keep our side, unchanged" too — finish such a path by hand (git merge origin/<base>, resolve, commit) and re-run merge-task, which then takes the clean path. A merge-flow node that stops for a human (an exchange-integrity or dangerous-diff stop) exits 2 the same way instead of escaping as a traceback. Gating is safety-only: running the command is the go-ahead (no review-state check), and the orchestrator never passes --admin, so branch protection / required checks remain the real gate. On any failure it runs git merge --abort, leaves the PR open, and exits non-zero — a DONE task is never downgraded. Flags mirror auto_merge: --strategy {merge,squash,rebase} (default git.auto_merge_strategy), --wait-for-checks/--no-wait-for-checks (default git.auto_merge_wait_for_checks), --no-resolve (abort on a conflict instead of running the flow). Like finalize/rerun it needs no live executor — the watch daemon or a worc run/rerun — because it works in the shared clone (--dry-run is exempt and names the owner). Both merge-task and prs --sync flip a depends_on dependency to merged, unblocking dependents.

Task dependencies (depends_on)#

A task can declare other tasks it needs merged first via front-matter depends_on: [<task-id>, …] (see task-authoring.md). Scheduling is non-blocking and merge-gated: under watch, a pending task is eligible only when every dependency has merged; while a dependency is unmerged the scheduler skips the dependent and runs other eligible pending tasks instead, so the single slot never idles on CI. The dependent is re-evaluated each tick (after the fetch/pull), and once its dependencies merge it branches from a base_branch that includes them. "Merged" is probed read-only (gh pr view → state == MERGED); a task that committed locally with no PR counts once it is terminal DONE. An open or armed PR (e.g. GitHub-native auto-merge still waiting on checks) is not yet merged. When that probe first observes a dependency MERGED, the daemon records the dependency's pr_merge audit op in state.db (idempotent, no network) even if the PR was merged out of band — so the merge-event audit ledger is complete for watch-driven merge-gated tasks without stopping the daemon for worc prs --sync.

Wait-forever policy + operator intervention. If a dependency is terminal-but-unmerged — it failed, went manual_action_required, or its PR was closed unmerged — the dependent is skipped every pass, indefinitely; the orchestrator never auto-fails it (an advisory log line task <id> waiting: … records each skip). To unblock, the operator either fixes/re-runs the dependency until it merges, or removes the depends_on entry (or the dependent task). A broken declaration — a cycle, a self-reference, or a depends_on id that matches no known task — is different: it is rejected fail-closed (terminal failed, invalid_depends_on, quarantined to .worc/tasks/rejected/), exactly like a malformed task. An explicit worc run <file> of a dependent whose dependencies are not merged is refused (non-zero exit) rather than skipped, so it never builds on a stale base.

Merge-SHA backfill. When the readiness probe observes a dependency's armed PR (merge_outcome: "armed", from GitHub-native auto-merge) now MERGED, it backfills the real mergeCommit.oid into that task's recorded merge outcome in state.db. The append-only ledger (completed.jsonl) keeps its point-in-time "armed" record — at terminal time the merge genuinely was only armed. An armed PR that no task depends on is never probed, so its recorded outcome stays "armed".

Managing the daemon (stop / restart)#

When watch runs as a background service (systemd, launchd, nohup &) you do not need to track its PID. A looping watch writes <repo>/.worc/orchestrator.pid on start and removes it on exit; two commands act on it from any shell in the same repo:

worc stop                  # ask the watcher to stop gracefully; idempotent
worc restart --poll-seconds 10   # stop the running watcher, then start a fresh loop with these flags

Shutdown works on Linux, macOS, and Windows. A soft stop is observed before the next flow node: the current node finishes, the next-node checkpoint is preserved, the task parks resumably, and the watcher removes its PID on exit. POSIX uses SIGTERM (followed by SIGCONT, so a suspended daemon actually receives it) plus the sentinel; Windows uses the sentinel because cross-shell os.kill is unavailable. If confirmation exceeds --timeout, neither platform escalates — the graceful stop stays pending (see the stop ladder above), and --force-full is the only immediate interrupt. The PID remains present until exit or hard kill is confirmed, so a second watcher cannot start and --force-full never loses its target. stop remains idempotent; on Windows an existing stop/children handle is preserved when PID evidence is missing because absence alone cannot prove the old watcher died.

Every command works under either name: this guide uses the short worc (worc watch, worc status, …); wastech-orchestrator is the equivalent canonical long form.