How-To#

Practical, problem-first recipes for situations you actually run into while operating the orchestrator — one problem, one fix, growing over time as new cases come up. For the full command reference behind each recipe, see Operations; for everyday setup and workflows, see the Cookbook.

1. The orchestrator failed and the git tree is dirty — resume from where it stopped#

Problem: A task run stopped with an error — worc status/worc list shows it as failed or manual_action_required — a crashed provider process, a rate limit, a missing tool on PATH, or a real quality issue you fixed by hand. The clone's working tree now has uncommitted changes left over from the interrupted attempt, and you don't want to lose that work or start the task over from scratch. You want to pick up exactly where it stopped.

Solution:

worc rerun <task-id> --continue

This resumes the task in place: the existing branch and all prior work are reused, and the pipeline re-enters at the stage that failed — the same recovery path crash recovery itself uses. If you'd rather see the plan first without touching anything:

worc rerun <task-id> --continue --dry-run

Details / caveats:

  • --continue needs an idle slot (no other active task) and the watch daemon stopped — it drives the pipeline directly in the shared clone.
  • Whether the dirty tree is tolerated depends on where the task stopped, and the rule is wider than it used to be: once a node that writes to the tree has run — an agent node, or any evaluator / checks / publish node downstream of one — the uncommitted changes are treated as the legitimate input to those nodes and are committed into the task automatically (you'll see a note: about it). So a run parked mid-implementation, before its first critic ever ran, resumes on the work it left on disk instead of asking you to throw it away. If it stopped before any writing node — at planning/refinement — a dirty tree is still refused as unaccounted work; resolve or stash it first.
  • To re-enter at a specific node instead of the recorded one (for example, re-running implementation after fixing something by hand), add --from <node>: for the default flow that is refinement, planning, implementation, testing, review, fixing, documentation, publish (a custom flow exposes its own node ids). This works even if the flow file itself changed since the checkpoint — the resume proceeds using whatever flow is currently on disk.

bash worc rerun task-001 --continue --from implementation

  • If the task had exhausted its fix-loop budget (manual_action_required, stuck at max_fix_cycles), resuming into that same loop asks interactively whether to reset it, e.g. review_fix (15/15) exhausted. Reset it to allow further fix rounds? [y/N]. Answer y, or pass --reset-fix-budget to skip the prompt in a script, or --no-reset-fix-budget to decline and refuse the resume instead. This prompt is never skipped by -y/--yes — resetting a fix budget is a consequential decision on purpose. A reset grants a fresh budget to the loop counters and to every non-blocking evaluator's max_rework_per_stage count; the global max_total_fix_iterations backstop is never reset.
  • A non-blocking evaluator (blocking: false) whose max_rework_per_stage is already spent never triggers that prompt — it would simply accept on its next pass, with no fix round, which is a full paid evaluator run that changes nothing. The plan names each such evaluator in a note: (max_rework_per_stage is spent for <node> (<used>/<cap>)); pass --reset-fix-budget if you want it to get fix rounds again.
  • Each --continue attempt appends a new ledger record linked to the failed one, so the retry history stays visible.

See Operations → Re-attempting a terminal task for the full set of rerun options, including the fresh, from-scratch mode.

2. A task failed with an error — how to re-run it#

Problem: worc status/worc list shows a task as failed (or manual_action_required), and you want to re-run it. The cause can be anything — a crashed provider, a rate limit, a missing tool, a real quality issue, or an environmental/transient problem (for example, you edited the flows while the daemon was serving the queue, so the flow file was momentarily absent when the daemon picked the task up: reason=unknown task_type 'implementation': no flow file implementation.yaml … Runworc install…). Once the underlying cause is fixed, you re-run — no need to recreate the task file.

First, decide which mode you need — fresh, restart-in-place, or --continue:

  • Fresh (default, worc rerun <task-id>): re-attempts the task from base as if new — resets the branch to base and runs the whole flow again. This is what a plain rerun does on a new-mode branch (the default), the only branch the orchestrator owns. Use it when the run never produced usable work — it died at pickup/flow-load or in an early stage (refinement/planning), or you simply want a clean attempt.
  • Restart in place (also just worc rerun <task-id> — chosen automatically): when the task runs on an operator-owned branch (branch_mode: existing/current) and failed before producing any work (no flow checkpoint — e.g. the flow-drift case above), a plain rerun must not reset a branch it doesn't own, so it re-drives the whole flow from the top on that branch as-is — nothing is reset and any commits already on the branch are kept. You pass no flag; rerun detects the case and the confirmation reads [restart] on branch '<branch>'. (If the operator-owned run did produce work, a fresh rerun refuses and points you at --continue instead.)
  • --continue: resumes in place — reuses the existing branch and re-enters at the stage that failed. Use this when the run already produced code you don't want to lose, which now includes a run parked mid-writing before any critic ran. This is section 1 above; see it for the dirty-tree and fix-budget details.

Not sure which? Preview without touching anything: worc rerun <task-id> --dry-run names the mode it would take (fresh / restart / continue) and writes nothing (add --continue to preview that mode).

Solution (normal, non-shell mode):

worc stop                          # rerun refuses while the watch daemon owns the clone; stop it first (a no-op if none runs)
worc rerun <task-id> --dry-run     # optional: preview the planned reconciliation, writes nothing
worc rerun <task-id>               # re-attempt (fresh, or restart-in-place on an operator branch); prompts y/N (add -y to skip)

Solution (inside worc shell):

down                               # stop the daemon — rerun is slot-guarded and refuses while it is up
rerun <task-id> --yes              # --yes is REQUIRED here: shell always runs rerun non-interactively, so an unconfirmed rerun is refused
up                                 # optional: resume serving the queue

Details / caveats:

  • Stop the daemon first. rerun drives the pipeline directly in the shared clone, so it refuses while a live watch daemon owns it (rerun: the watch daemon is running (pid …); stop it first). Use worc stop (non-shell) or down (shell).
  • In worc shell, always pass --yes. The console runs rerun non-interactively (a confirmation prompt would fight the REPL's own stdin reader), so a bare rerun <task-id> is refused with "refusing without confirmation (non-interactive)". The finalize and merge-task verbs are slot-guarded the same way.
  • You don't move the task file. rerun finds it automatically wherever it currently lives (tasks/pending/, tasks/done/, or tasks/failed/); leave the failed file in tasks/failed/.
  • Restart-in-place keeps the branch and the attempt history. For a pre-checkpoint failure on an operator-owned branch, use rerun (not a manual re-queue via worc run <file>): it re-drives on the branch without resetting it and records the retry in the ledger as attempt 2 (rerun_of set), so the failure → retry chain stays auditable. A manual re-queue loses that linkage.
  • Run it from the target repo (the one holding .worc/), not from the orchestrator repo. worc is the short alias for wastech-orchestrator — the commands are identical.
  • Get the exact id with worc list --format ids --scope rerun (lists only the failed / manual_action_required ids a rerun accepts).
  • Avoid the cause: don't edit .worc/flows/ while the daemon is serving the queue — stop it (down / worc stop), change the flows, then start again, so a task can't be picked up mid-edit.

3. A task was stopped but still holds the slot (parked (no daemon)) — free the stuck slot#

Problem: You stopped a run mid-task (a down in worc shell, a worc stop, a --force-full, or the process just died), and now the task still owns the one processing slot even though nothing is executing, so starting anything else fails with "a task is active". worc stop/down prints a note about this and points at the fix; worc status/top/list show the task as parked (no daemon); and a new worc run names the blocking task in its refusal. If the process simply died (no clean stop), you may not have seen the note — the parked (no daemon) label is the tell.

Why this happens: stop/down/--force-full stop the daemon, not the task. By the recovery model, the single unfinished task is deliberately left running at its checkpoint so the next up/watch can resume it — so a running row with no live executor — neither a daemon nor a run/rerun in flight, each of which records itself in .worc/orchestrator.pid / .worc/orchestrator.run — means "parked, awaiting resume", not "executing now" (which is exactly what parked (no daemon) reports). That parked row still holds the one processing slot, which is why a new task is refused. stop has no authority over a task row, so it can never clear it — it's the wrong tool for that, even though it now tells you which tool is right.

Solution — first confirm nothing is actually running, then pick one of three:

# 1. Confirm there's no live daemon and no agent process (from the target repo):
worc status                                   # shows `parked (no daemon)` + the stuck task id + node
ls .worc/*.pid 2>/dev/null                     # no output = no daemon
ps aux | grep -iE "wastech|worc|codex|claude" | grep -v grep   # empty = no agent running

Then, depending on what you want the task to do:

# A. Close it (you're done with this task / it's superseded):
worc finalize <task-id> --as failed -y         # records terminal, checks out base, frees the slot

# B. Continue it from where it stopped (you still want it finished):
worc rerun <task-id> --continue                 # resumes in place at the checkpoint

# C. Just let it finish (restart the daemon — resume() picks it up first):
worc watch                                      # or `up` in the shell

Details / caveats:

  • Verify "nothing is running" before finalizing. If a worc run/daemon really is still alive in another terminal, don't finalize under it — stop that first (worc stop). The ls .worc/*.pid + ps checks above are the tell: no PID file and no worc/codex/claude process means the running is stale.
  • finalize needs an idle slot and no live daemon — it runs terminal cleanup in the shared clone (git checkout base when the branch mode / repo.checkout_base_on_cleanup calls for it). With the daemon already stopped this is satisfied. --as failed keeps the task rerun-eligible; --as abandoned records it as manually abandoned.
  • Preview first if unsure: worc finalize <task-id> --as failed --dry-run prints exactly what it will do (status transition, whether cleanup checks out base or stays on the branch, ledger record) and writes nothing.
  • finalize leaves an operator-owned branch alone — it never deletes or resets a branch_mode: existing/current branch, and by default it does not even check out base for those modes (they stay on the branch). Only new mode returns to base by default; set repo.checkout_base_on_cleanup to override either way.
  • The fix is finalize/rerun/restart — never stop. stop only manages the daemon; reaching for it here is the natural mistake this recipe exists to correct.

4. Track your operator flows (.worc/flows/) in git#

Problem: install gitignores the whole .worc/ runtime home as one unit — state.db, logs/, workspace/, config.yaml, and your editable flows/ copies all disappear from git status together (see Operations → Installation). But .worc/flows/ holds the actual behavior you hand-author: the flow YAML files and their prompts (see Flow authoring). By default that content has no git history, produces no diff to review, and can't be shared with teammates or shipped through a PR — you have to pass files around some other way.

Solution: replace the blanket .worc/ line in the repo's tracked .gitignore with a wildcard pattern that ignores everything under .worc/ except flows/:

# Ignore the runtime home's contents (not the dir itself) so flows/ can be re-included:
# Git won't descend into a fully-excluded dir, so `.worc/` would make any !.worc/flows a no-op.
.worc/*
!.worc/flows/

Then track and commit the flows you want to keep:

git add .gitignore .worc/flows
git commit -m "chore: track .worc/flows in git"

Details / caveats:

  • Replace the line, don't add to it. install (and install --reconfigure) append a single blanket .worc/ line. Keep only one scheme — a blanket .worc/ and a .worc/* + !.worc/flows/ pair are mutually exclusive; the blanket form always wins if both are present (see below), so delete it once you switch.
  • Why the wildcard is needed, not just !.worc/flows/: Git never descends into a directory that's already excluded, so a negation for a path inside an excluded directory is a no-op — this is the classic gitignore gotcha the comment calls out. .worc/* ignores each direct child individually (so git still walks into .worc/), which lets !.worc/flows/ re-include that one child and everything nested under it (the per-flow prompt folders, roles/supervisor.md, etc.) without an extra pattern per file.
  • Everything else under .worc/ stays ignored. state.db, logs/ (check logs included, under logs/<task-id>/checks/), runs/, workspace/, config.yaml, tools/, and the memory home never ride along just because flows/ is now tracked.
  • The orchestrator won't fight you. A parent-directory exclusion from any source — tracked .gitignore, the untracked clone-local .git/info/exclude, or the global excludes file — blocks re-inclusion of its children, regardless of which file or line added it. Two mechanisms could otherwise reintroduce a blanket .worc/ line and silently defeat your !.worc/flows/ negation: the per-task-run safety net (ensure_runtime_excludes()) and install --reconfigure's .gitignore writer. Both check whether a runtime-only path (.worc/state.db) is already ignored before appending anything, so once you've switched to the wildcard scheme they leave it alone.
  • Same trick works for other subdirectories — e.g. add !.worc/tools/ if you also want the packaged tool executables tracked.

See also: flow-authoring.md for where flows and prompts live, and operations → Installation for what install writes into .worc/ by default.

5. Fix conflicting Codex installations on Windows#

Problem: codex --version in a terminal reports one version, while worc preflight reports another. A real task may then fail every shell command with an infrastructure error such as CreateProcessWithLogonW failed: 2, windows sandbox failed, or helper copy failed for command-runner, even though codex sandbox ... works when you run it by hand.

Why this happens: Codex can be installed by more than one independently updated surface: a global npm package, a Node version manager, the Windows Codex app, or an IDE integration. On Windows, a global npm install normally puts codex.cmd on PATH, while an app-managed standalone package may expose a different codex.exe. A terminal can select the .cmd launcher through PATHEXT; the orchestrator deliberately starts providers as an argv list without a shell, so the Windows process launcher can instead select the later .exe. The CLI version, sandbox setup helper, and codex-command-runner.exe may then come from different package roots.

Multiple installations are not inherently broken. The ambiguous bare command is the problem.

Diagnose it from the same Windows account that runs worc:

where.exe codex
Get-Command codex -All | Format-Table CommandType, Name, Source
codex --version
worc preflight

# Optional: compare the Windows app and its app-managed standalone CLI.
Get-AppxPackage OpenAI.Codex | Select-Object Name, Version, Status
Get-Content "$env:USERPROFILE\.codex\packages\standalone\current\codex-package.json"

More than one where.exe result, or different versions from codex --version and worc preflight, confirms ambiguous resolution. For a global npm install, locate its physical native Windows executable instead of the top-level codex.cmd launcher:

$npmRoot = npm.cmd root -g
$codexExe = Join-Path $npmRoot `
  '@openai\codex\node_modules\@openai\codex-win32-x64\vendor\x86_64-pc-windows-msvc\bin\codex.exe'

Test-Path -LiteralPath $codexExe
& $codexExe --version

The exact package root may differ under nvm, pnpm, Yarn, or another CPU architecture. If the path above is absent, find the manager-owned native executable and verify that its package also contains the matching resources:

<package-root>\bin\codex.exe
<package-root>\codex-resources\codex-windows-sandbox-setup.exe
<package-root>\codex-resources\codex-command-runner.exe

Solution: pin that physical executable in the target repository's .worc/config.yaml. Use single-quoted YAML on Windows so backslashes remain literal:

agents:
  providers:
    codex:
      command: 'C:\absolute\package-root\bin\codex.exe'

Then restart worc watch if it is running and verify both provider discovery and a real sandboxed process:

worc preflight

$pwsh = (Get-Command pwsh.exe -CommandType Application | Select-Object -First 1).Source
& $codexExe sandbox $pwsh -NoProfile -Command 'Write-Output OK'

The preflight version must now match & $codexExe --version, its Windows sandbox helper path must belong to the same installation, and the smoke command must print OK with exit code 0.

Do not:

  • point command at codex.cmd; it requires a command shell, which the orchestrator intentionally does not use;
  • copy sandbox helpers between releases or manually retarget the app-managed current junction;
  • delete an old standalone release before pinning and verifying the replacement — the Windows app may own hardlinks or junctions into that release, and deleting it does not make worc select npm;
  • weaken workspace-write, approvals, or security.strict_isolation to hide a packaging/path problem.

If you do not use the Windows Codex app, uninstall it through Windows Settings / Microsoft Store only after the pinned npm executable passes the checks above. If you keep the app, it is safe to leave its standalone CLI installed: the absolute command removes the ambiguity, and each updater can continue managing its own files.

The same principle applies on Linux and macOS when Homebrew, npm, a version manager, and an IDE expose different Codex installations: compare type -a codex / which -a codex with worc preflight, then configure the absolute path to the intended executable. A POSIX launcher with a valid shebang can run without a shell; the Windows-specific requirement is to avoid .cmd and use the native .exe.

For the supported native Windows sandbox modes and upstream troubleshooting guidance, see the Codex Windows sandbox documentation.

6. Track your task files (tasks/) in git#

Problem: install gitignores the task lifecycle tree — tasks/preparing/, tasks/pending/, tasks/done/, tasks/failed/ — so your task files never appear in git status, never ride a task's branch or its pull request, and no audit commit is made. That is the right default for one operator on one machine, but it rules out three things you may want: a task file reviewed in a pull request before it runs, a task handed to a teammate (or to a daemon on another machine) by committing and pushing it, and a permanent audit trail of what was asked for, committed next to the change that answered it.

Solution — at install time, answer yes to the wizard's question, or pass the flag:

worc install . --track-tasks

Or on an existing install, delete the one line install seeded in the repo's tracked .gitignore:

# wastech-orchestrator task lifecycle (auto-appended by `worc install`; delete this line to track tasks in git)
/tasks/

Then track what you want to keep:

git add .gitignore tasks/
git commit -m "chore: track the task lifecycle in git"

What changes once the tree is tracked:

  • Each terminal outcome makes an audit commit. The task file and its <task-id>.summary.md are committed on the task's branch (git.footprint.audit_commit_message, git.footprint.audit_on_branch) — separately from the code commit, and scoped to that one task, so a task pending beside it is never swept in.
  • git pull becomes a queue. A watch daemon fetches your base branch each tick, so a task file committed and pushed by anyone is picked up by whichever machine is watching. This is the only way to hand a task over through git.
  • Drafts become visible. A half-written file in tasks/preparing/ shows up in git status like any other new file. The watcher never scans preparing/, so it is still safe to compose there — it is only your working tree that is no longer clean.
  • One thing looks like a bug the first time. With audit_on_branch: task (the default) the move into tasks/done/ is committed on the task branch, while your own commit of the file sits on base — in tasks/pending/, or tasks/preparing/ if you committed it there. Terminal cleanup returns the tree to base and git restores that copy, so the finished task's file reappears where you committed it and stays until the pull request merges. Expected: the orchestrator recognises its own finished task by content — the daemon skips it, and worc run on it answers duplicate_task_id without moving or deleting anything.

Details / caveats:

  • Delete the line, don't negate it. A !tasks/ negation after a /tasks/ exclusion is a no-op — git never descends into an excluded directory — and there is nothing under the tree you would want to keep ignored. Remove the line (and its comment).
  • Nothing re-adds it behind your back. Only install writes that line, and it skips it in two cases: when the tree is already ignored, and when git already tracks something under it — which is exactly the state you are in after committing your task files, so a plain re-run and install --reconfigure both leave your choice alone and say left tasks/ tracked (git already tracks files there). The per-run safety net that repairs ignore rules in the clone-local .git/info/exclude deliberately covers only .worc/ and .worc-io/, never the lifecycle tree — an operator cannot be expected to find a rule in a file they never open.
  • Delete the line before you commit anything, though. The skip above is driven by tracked content, so between deleting the line and making that first git add, a --reconfigure would put it back. Do the two steps together.
  • If you renamed paths.tasks_dir, the line names the old directory. The orchestrator asks git about the directory in your config, so a stale /tasks/ line leaves the renamed tree tracked whether you wanted that or not. Move the line with the rename (and create the four lifecycle subfolders yourself — install scaffolds only the directory it was given).
  • The reverse direction works the same way. To stop tracking a tree you already track, add the line back — and remember that an ignore rule does not untrack: git rm -r --cached tasks/ and a commit are what remove the files from the index.
  • Already-committed task files keep working after you switch to the default. A tracked file under an ignored tree is still committed when a lifecycle move deletes it, so no dangling deletion is left on your base branch; the newly moved copies simply stop being added.
  • What the orchestrator records either way. Tracking task files is a convenience, not the record: state.db, logs/completed.jsonl, logs/<task-id>/, and the <task-id>.summary.md beside the task file are written whether or not git sees any of it.

See also: Configuration → paths for the key itself, Operations → Installation for what install writes, and section 4 above for the same question asked of .worc/flows/.