Stop workspace before deleting — Coder returns 405 on running workspaces

The DELETE API requires the workspace to be stopped first. Now stops,
waits 10s for shutdown, then deletes. This was causing workspaces to
never be cleaned up after task completion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

README.md

@@ -1,39 +1,184 @@
 # SDLC Orchestrator
 
-Lightweight webhook-driven orchestrator that replaces n8n for the Gitea + Coder + Claude Code SDLC pipeline.
+Lightweight webhook-driven orchestrator for the Gitea + Coder + Claude Code SDLC pipeline. Receives Gitea webhooks, creates ephemeral Coder workspaces running Claude Code, and cleans up when done.
-
-Receives Gitea webhooks, routes them to the correct SDLC stage, creates Coder workspaces, and posts status comments back on issues/PRs.
 
 ## Architecture
 
 ```
-Gitea Webhooks → SDLC Orchestrator → Coder API (creates workspace)
+Gitea Webhook → Orchestrator → Coder API (create workspace)
-                                   → Gitea API (posts comments)
+                             → Gitea API (post status comment)
+
+Workspace runs Claude Code → reads issue → does work → posts results to Gitea
+                           → POST /webhook/task-complete/:name (callback)
+
+Orchestrator receives callback → deletes workspace via Coder API
 ```
 
-~500 lines of TypeScript. No database, no UI, no state — just a webhook router.
+## Adding a New Project
 
-## Webhook Endpoints
+To add a new Gitea repo to the SDLC pipeline:
 
-| Endpoint | Gitea Event | Action |
+### 1. Add Claude Code slash commands to the repo
-|----------|-------------|--------|
-| `POST /webhook/gitea-issue-triage` | Issue opened | Route to `analyse` or `fix-bug` |
-| `POST /webhook/gitea-issue-label` | Issue labeled | Stage transition (architect/develop/test/devops) |
-| `POST /webhook/gitea-issue-comment` | Comment created | Re-trigger analyst if still in analysis |
-| `POST /webhook/gitea-pr-review` | PR opened/synced | Trigger code review |
-| `POST /webhook/gitea-pr-review-rework` | Review submitted | Route to `rework-pr` or `test` |
-| `POST /webhook/gitea-release` | Release issue | Tag RC or production release |
-| `GET /health` | — | Health check |
 
-Plus a cron job (Monday 9 AM) for weekly maintenance.
+Create `.claude/commands/` in the repo with command files for each SDLC stage:
 
-## Setup
+```
+.claude/commands/
+├── analyse.md        # Business analyst — requirements & acceptance criteria
+├── architect.md      # Systems architect — design doc & spec
+├── develop.md        # Developer — implement & create PR
+├── review-code.md    # Reviewer — approve or request changes
+├── test.md           # QA — test report & additional tests
+├── fix-bug.md        # Bug fix — skip analyse/architect
+├── rework-pr.md      # Address review feedback
+├── release.md        # Tag RC or production release
+├── devops.md         # Migration & deployment
+└── maintenance.md    # Weekly maintenance tasks
+```
+
+Each file uses this format:
+
+```markdown
+---
+description: What this command does
+allowed-tools: Read, Bash, Glob, Grep
+---
+
+You are a **Role Name**. Your job is to...
+
+## Process
+
+1. **Read the issue**:
+   ```bash
+   curl -s "https://gitea.samson.media/api/v1/repos/${GITEA_ORG}/${GITEA_REPO}/issues/$ARGUMENTS" \
+     -H "Authorization: token ${GITEA_TOKEN}" | jq '{title, body, labels: [.labels[].name]}'
+   ```
+
+2. **Do your work...**
+
+3. **Post a comment**:
+   ```bash
+   curl -X POST "https://gitea.samson.media/api/v1/repos/${GITEA_ORG}/${GITEA_REPO}/issues/$ARGUMENTS/comments" \
+     -H "Authorization: token ${GITEA_TOKEN}" \
+     -H "Content-Type: application/json" \
+     -d '{"body": "..."}'
+   ```
+```
+
+`$ARGUMENTS` is replaced with the issue/PR number at runtime.
+
+### 2. Add bot users as collaborators
+
+Add both `claude-dev` and `claude-review` as **direct collaborators** with **Write** access on the repo. Org membership alone is not sufficient.
+
+### 3. Configure Gitea webhooks
+
+In the repo's **Settings → Webhooks**, create these webhooks:
+
+| # | URL | Events |
+|---|-----|--------|
+| 1 | `https://sdlc.samson.media/webhook/gitea-issue-triage` | Issues: `opened` |
+| 2 | `https://sdlc.samson.media/webhook/gitea-issue-label` | Issues: `labeled` |
+| 3 | `https://sdlc.samson.media/webhook/gitea-issue-comment` | Issue Comment: `created`, `edited` |
+| 4 | `https://sdlc.samson.media/webhook/gitea-pr-review` | Pull Request: `opened`, `synchronized` |
+| 5 | `https://sdlc.samson.media/webhook/gitea-pr-review-rework` | Pull Request Review: `submitted` |
+| 6 | `https://sdlc.samson.media/webhook/gitea-release` | Issues: `opened`, `labeled` |
+
+All webhooks use:
+- Content type: `application/json`
+- Method: `POST`
+
+### 4. (Optional) Add to maintenance cron
+
+To include the repo in weekly maintenance, add it to the `MAINTENANCE_REPOS` env var:
+
+```
+MAINTENANCE_REPOS=org1/repo1=https://gitea.samson.media/org1/repo1.git,org2/repo2=https://gitea.samson.media/org2/repo2.git
+```
+
+### 5. Test it
+
+Create an issue in the repo. The orchestrator will:
+1. Receive the `issues/opened` webhook
+2. Create a Coder workspace running `/analyse`
+3. Claude reads the issue, posts clarifying questions as a comment
+4. Workspace is deleted automatically when done
+
+## SDLC Flow
+
+| Stage | Trigger | Task Type | Bot Account |
+|-------|---------|-----------|-------------|
+| Analyse | Issue opened / comment | `analyse` | claude-dev |
+| Architect | Label: `ready-for-architecture` | `architect` | claude-dev |
+| Develop | Label: `ready-for-development` | `develop` | claude-dev |
+| Review | PR opened/updated | `review-code` | claude-review |
+| Rework | Review: changes requested | `rework-pr` | claude-dev |
+| Test | Review: approved | `test` | claude-review |
+| Deploy | Label: `ready-for-deployment` | `devops` | claude-dev |
+| Fix Bug | Issue opened with `bug` label | `fix-bug` | claude-dev |
+
+## Infrastructure Setup
 
 ### Prerequisites
 
-- Node.js 22+
+- k3s cluster with Traefik ingress and cert-manager
-- Coder running with the `cloudflare-worker` template (see `coder/`)
+- Coder instance (e.g. `coder.samson.media`)
-- Gitea with two bot accounts: `claude-dev` and `claude-review`
+- Gitea instance (e.g. `gitea.samson.media`)
+- Docker registry (e.g. `registry.samson.media`)
+- Claude Code Max subscription (for OAuth token)
+
+### Deploy the Orchestrator
+
+1. **Build the workspace image** (pre-installed Node.js, Claude Code, Playwright deps):
+   ```bash
+   docker build -f coder/workspace.Dockerfile -t registry.samson.media/coder-workspace:latest .
+   docker push registry.samson.media/coder-workspace:latest
+   ```
+
+2. **Push the Coder template**:
+   ```bash
+   coder templates push cloudflare-worker --directory coder/cloudflare-worker --yes
+   ```
+   Note the template ID from the Coder dashboard.
+
+3. **Create Gitea bot accounts**:
+   - `claude-dev` — used for most stages (analyse, develop, etc.)
+   - `claude-review` — used for review and test stages
+   - Generate API tokens for each
+
+4. **Generate Claude Code OAuth token** (uses Max subscription instead of API credits):
+   ```bash
+   claude setup-token
+   ```
+   This opens a browser for auth and outputs a long-lived token (`sk-ant-oat01-...`).
+
+5. **Create k8s secrets**:
+   ```bash
+   kubectl create namespace sdlc-orchestrator
+   kubectl create secret generic orchestrator-secrets -n sdlc-orchestrator \
+     --from-literal=CODER_URL=https://coder.samson.media \
+     --from-literal=CODER_TOKEN=<coder-session-token> \
+     --from-literal=CODER_TEMPLATE_ID=<template-id> \
+     --from-literal=GITEA_DEV_TOKEN=<claude-dev-token> \
+     --from-literal=GITEA_REVIEW_TOKEN=<claude-review-token> \
+     --from-literal=CLAUDE_OAUTH_TOKEN=<oauth-token-from-step-4> \
+     --from-literal=CALLBACK_URL=https://sdlc.samson.media \
+     --from-literal=GITEA_URL=https://gitea.samson.media \
+     --from-literal=BOT_DEV_USERNAME=claude-dev \
+     --from-literal=BOT_REVIEW_USERNAME=claude-review
+   ```
+
+6. **Deploy**:
+   ```bash
+   # Tag to trigger CI build
+   git tag -a v1.0.0 -m "Initial release"
+   git push origin main --tags
+
+   # Or deploy manually
+   docker build -t registry.samson.media/sdlc-orchestrator:latest .
+   docker push registry.samson.media/sdlc-orchestrator:latest
+   kubectl apply -f k8s/deployment.yaml
+   ```
 
 ### Environment Variables
 
@@ -43,58 +188,58 @@ Plus a cron job (Monday 9 AM) for weekly maintenance.
 | `CODER_URL` | Yes | Coder API base URL |
 | `CODER_TOKEN` | Yes | Coder session token |
 | `CODER_TEMPLATE_ID` | Yes | Coder workspace template ID |
-| `GITEA_DEV_TOKEN` | Yes | Gitea API token for `claude-dev` |
+| `GITEA_DEV_TOKEN` | Yes | Gitea API token for claude-dev |
-| `GITEA_REVIEW_TOKEN` | Yes | Gitea API token for `claude-review` |
+| `GITEA_REVIEW_TOKEN` | Yes | Gitea API token for claude-review |
-| `ANTHROPIC_API_KEY` | Yes | Anthropic API key for Claude Code |
+| `CLAUDE_OAUTH_TOKEN` | Yes | Claude Code Max OAuth token |
+| `CALLBACK_URL` | Yes | Public URL of this service (e.g. `https://sdlc.samson.media`) |
 | `GITEA_URL` | No | Gitea base URL (default: `https://gitea.samson.media`) |
 | `BOT_DEV_USERNAME` | No | Dev bot username (default: `claude-dev`) |
 | `BOT_REVIEW_USERNAME` | No | Review bot username (default: `claude-review`) |
+| `QUEUE_CONCURRENCY` | No | Max concurrent workspaces (default: 2) |
 | `MAINTENANCE_REPOS` | No | Comma-separated `org/repo=clone_url` for weekly maintenance |
 
-### Local Development
+> **Important**: Do NOT set `ANTHROPIC_API_KEY` in the workspace — it takes priority over the OAuth token and will use pay-per-use credits instead of your Max subscription.
 
-```bash
+## Webhook Endpoints
-npm install
-cp k8s/secret.yaml.example .env  # Edit with real values (use KEY=value format)
-npm run dev
-```
 
-### Deploy to k3s
+| Endpoint | Purpose |
+|----------|---------|
+| `GET /health` | Health check |
+| `GET /queue` | Queue status (pending, running, active workspaces) |
+| `POST /webhook/gitea-issue-triage` | Route new issues to analyse or fix-bug |
+| `POST /webhook/gitea-issue-label` | Stage transitions via labels |
+| `POST /webhook/gitea-issue-comment` | Re-trigger analyst on new/edited comments |
+| `POST /webhook/gitea-pr-review` | Trigger code review on PR open/update |
+| `POST /webhook/gitea-pr-review-rework` | Route review outcomes to rework or test |
+| `POST /webhook/gitea-release` | Handle release workflow |
+| `POST /webhook/task-complete/:name` | Callback from workspaces when done |
 
-```bash
+## How Workspaces Work
-# Build and push image
-docker build -t registry.samson.media/sdlc-orchestrator:latest .
-docker push registry.samson.media/sdlc-orchestrator:latest
 
-# Create secrets
+Workspaces are **ephemeral** (emptyDir, no persistent storage):
-cp k8s/secret.yaml.example k8s/secret.yaml
-# Edit k8s/secret.yaml with real values
-kubectl apply -f k8s/secret.yaml
 
-# Deploy
+1. Orchestrator creates workspace via Coder API with parameters (issue number, task type, tokens, callback URL)
-kubectl apply -f k8s/deployment.yaml
+2. Startup script clones the repo and checks out `develop`
-```
+3. For heavy stages (develop, test, review, rework, fix-bug, devops): installs deps, builds frontend, installs Playwright
+4. For lightweight stages (analyse, architect, release, maintenance): skips build
+5. Reads `.claude/commands/<task_type>.md`, strips YAML frontmatter, substitutes `$ARGUMENTS` with issue number
+6. Runs `claude -p --dangerously-skip-permissions --verbose "<prompt>"`
+7. On completion (success or failure via ERR/EXIT trap): POSTs to callback URL
+8. Orchestrator receives callback, deletes workspace via Coder API
 
-### Configure Gitea Webhooks
+## Resilience
 
-For each project, add these webhooks in **Settings → Webhooks**:
+- **Deduplication**: Tasks keyed by `{taskType}-{repo}-{issue}`. Duplicate webhooks are dropped.
+- **Startup reconciliation**: On boot, queries Coder for running workspaces and re-adopts them into the queue.
+- **409 handling**: If workspace already exists — adopts if running, deletes and retries if stopped/failed.
+- **Callback trap**: Startup script uses `trap` to always fire the callback, even on clone failure or other errors.
+- **TTL safety net**: Coder template has 1-hour auto-stop as a backstop for missed callbacks.
 
-| Event | URL |
+## CI/CD
-|-------|-----|
-| Issues (opened) | `https://sdlc.samson.media/webhook/gitea-issue-triage` |
-| Issues (labeled) | `https://sdlc.samson.media/webhook/gitea-issue-label` |
-| Issue Comments (created) | `https://sdlc.samson.media/webhook/gitea-issue-comment` |
-| Pull Request (opened, synchronized) | `https://sdlc.samson.media/webhook/gitea-pr-review` |
-| Pull Request Review (submitted) | `https://sdlc.samson.media/webhook/gitea-pr-review-rework` |
-| Issues (opened, labeled) — releases | `https://sdlc.samson.media/webhook/gitea-release` |
 
-## Coder Template
+| Workflow | Trigger | Output |
+|----------|---------|--------|
+| `publish.yml` | `v*` tags | `registry.samson.media/sdlc-orchestrator:<version>` + `:latest` |
+| `publish-workspace.yml` | Changes to `coder/workspace.Dockerfile` on main | `registry.samson.media/coder-workspace:latest` |
 
-The `coder/` directory contains the Terraform template for ephemeral Kubernetes workspaces. See `coder/README.md`.
+Fleet GitOps (in `samson-media/devops` repo) watches `registry.samson.media/sdlc-orchestrator:latest` for deployment.
-
-## Migrating from n8n
-
-1. Deploy this service to k3s
-2. Update Gitea webhooks to point to `sdlc.samson.media` instead of `n8n.samson.media`
-3. Verify with a test issue
-4. Decommission n8n

coder/cloudflare-worker/main.tf

@@ -423,7 +423,6 @@ resource "coder_agent" "main" {
   }
 
   env = {
-    ANTHROPIC_API_KEY        = data.coder_parameter.anthropic_api_key.value
     CLAUDE_CODE_OAUTH_TOKEN  = data.coder_parameter.claude_oauth_token.value
     GITHUB_TOKEN      = data.coder_external_auth.github.access_token
     GITEA_TOKEN       = data.coder_parameter.gitea_token.value
@@ -581,11 +580,6 @@ resource "kubernetes_deployment_v1" "workspace" {
             value = coder_agent.main.token
           }
 
-          env {
-            name  = "ANTHROPIC_API_KEY"
-            value = data.coder_parameter.anthropic_api_key.value
-          }
-
           env {
             name  = "CLAUDE_CODE_OAUTH_TOKEN"
             value = data.coder_parameter.claude_oauth_token.value

src/queue.ts

@@ -127,10 +127,14 @@ export class TaskQueue {
     this.active.delete(workspaceName);
 
     try {
+      await this.coder.stopWorkspace(entry.workspaceId);
+      console.log(`[queue] workspace stopped: ${workspaceName} — waiting for shutdown`);
+      // Wait for the workspace to fully stop before deleting
+      await new Promise((resolve) => setTimeout(resolve, 10_000));
       await this.coder.deleteWorkspace(entry.workspaceId);
       console.log(`[queue] workspace deleted: ${workspaceName}`);
     } catch (err) {
-      console.error(`[queue] failed to delete workspace ${workspaceName}:`, err);
+      console.error(`[queue] failed to clean up workspace ${workspaceName}:`, err);
     }
 
     // Drain in case pending tasks were waiting for capacity