Fix BullMQ jobId format — colons are not allowed

BullMQ throws "Custom Id cannot contain :" when jobId includes colons.
Replace colon separators with dashes in all jobId values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

README.md

@@ -1,39 +1,184 @@
 # SDLC Orchestrator
 
-Lightweight webhook-driven orchestrator that replaces n8n for the Gitea + Coder + Claude Code SDLC pipeline.
+Lightweight webhook-driven orchestrator for the Gitea + Coder + Claude Code SDLC pipeline. Receives Gitea webhooks, creates ephemeral Coder workspaces running Claude Code, and cleans up when done.
-
-Receives Gitea webhooks, routes them to the correct SDLC stage, creates Coder workspaces, and posts status comments back on issues/PRs.
 
 ## Architecture
 
 ```
-Gitea Webhooks → SDLC Orchestrator → Coder API (creates workspace)
+Gitea Webhook → Orchestrator → Coder API (create workspace)
-                                   → Gitea API (posts comments)
+                             → Gitea API (post status comment)
+
+Workspace runs Claude Code → reads issue → does work → posts results to Gitea
+                           → POST /webhook/task-complete/:name (callback)
+
+Orchestrator receives callback → deletes workspace via Coder API
 ```
 
-~500 lines of TypeScript. No database, no UI, no state — just a webhook router.
+## Adding a New Project
 
-## Webhook Endpoints
+To add a new Gitea repo to the SDLC pipeline:
 
-| Endpoint | Gitea Event | Action |
+### 1. Add Claude Code slash commands to the repo
-|----------|-------------|--------|
-| `POST /webhook/gitea-issue-triage` | Issue opened | Route to `analyse` or `fix-bug` |
-| `POST /webhook/gitea-issue-label` | Issue labeled | Stage transition (architect/develop/test/devops) |
-| `POST /webhook/gitea-issue-comment` | Comment created | Re-trigger analyst if still in analysis |
-| `POST /webhook/gitea-pr-review` | PR opened/synced | Trigger code review |
-| `POST /webhook/gitea-pr-review-rework` | Review submitted | Route to `rework-pr` or `test` |
-| `POST /webhook/gitea-release` | Release issue | Tag RC or production release |
-| `GET /health` | — | Health check |
 
-Plus a cron job (Monday 9 AM) for weekly maintenance.
+Create `.claude/commands/` in the repo with command files for each SDLC stage:
 
-## Setup
+```
+.claude/commands/
+├── analyse.md        # Business analyst — requirements & acceptance criteria
+├── architect.md      # Systems architect — design doc & spec
+├── develop.md        # Developer — implement & create PR
+├── review-code.md    # Reviewer — approve or request changes
+├── test.md           # QA — test report & additional tests
+├── fix-bug.md        # Bug fix — skip analyse/architect
+├── rework-pr.md      # Address review feedback
+├── release.md        # Tag RC or production release
+├── devops.md         # Migration & deployment
+└── maintenance.md    # Weekly maintenance tasks
+```
+
+Each file uses this format:
+
+```markdown
+---
+description: What this command does
+allowed-tools: Read, Bash, Glob, Grep
+---
+
+You are a **Role Name**. Your job is to...
+
+## Process
+
+1. **Read the issue**:
+   ```bash
+   curl -s "https://gitea.samson.media/api/v1/repos/${GITEA_ORG}/${GITEA_REPO}/issues/$ARGUMENTS" \
+     -H "Authorization: token ${GITEA_TOKEN}" | jq '{title, body, labels: [.labels[].name]}'
+   ```
+
+2. **Do your work...**
+
+3. **Post a comment**:
+   ```bash
+   curl -X POST "https://gitea.samson.media/api/v1/repos/${GITEA_ORG}/${GITEA_REPO}/issues/$ARGUMENTS/comments" \
+     -H "Authorization: token ${GITEA_TOKEN}" \
+     -H "Content-Type: application/json" \
+     -d '{"body": "..."}'
+   ```
+```
+
+`$ARGUMENTS` is replaced with the issue/PR number at runtime.
+
+### 2. Add bot users as collaborators
+
+Add both `claude-dev` and `claude-review` as **direct collaborators** with **Write** access on the repo. Org membership alone is not sufficient.
+
+### 3. Configure Gitea webhooks
+
+In the repo's **Settings → Webhooks**, create these webhooks:
+
+| # | URL | Events |
+|---|-----|--------|
+| 1 | `https://sdlc.samson.media/webhook/gitea-issue-triage` | Issues: `opened` |
+| 2 | `https://sdlc.samson.media/webhook/gitea-issue-label` | Issues: `labeled` |
+| 3 | `https://sdlc.samson.media/webhook/gitea-issue-comment` | Issue Comment: `created`, `edited` |
+| 4 | `https://sdlc.samson.media/webhook/gitea-pr-review` | Pull Request: `opened`, `synchronized` |
+| 5 | `https://sdlc.samson.media/webhook/gitea-pr-review-rework` | Pull Request Review: `submitted` |
+| 6 | `https://sdlc.samson.media/webhook/gitea-release` | Issues: `opened`, `labeled` |
+
+All webhooks use:
+- Content type: `application/json`
+- Method: `POST`
+
+### 4. (Optional) Add to maintenance cron
+
+To include the repo in weekly maintenance, add it to the `MAINTENANCE_REPOS` env var:
+
+```
+MAINTENANCE_REPOS=org1/repo1=https://gitea.samson.media/org1/repo1.git,org2/repo2=https://gitea.samson.media/org2/repo2.git
+```
+
+### 5. Test it
+
+Create an issue in the repo. The orchestrator will:
+1. Receive the `issues/opened` webhook
+2. Create a Coder workspace running `/analyse`
+3. Claude reads the issue, posts clarifying questions as a comment
+4. Workspace is deleted automatically when done
+
+## SDLC Flow
+
+| Stage | Trigger | Task Type | Bot Account |
+|-------|---------|-----------|-------------|
+| Analyse | Issue opened / comment | `analyse` | claude-dev |
+| Architect | Label: `ready-for-architecture` | `architect` | claude-dev |
+| Develop | Label: `ready-for-development` | `develop` | claude-dev |
+| Review | PR opened/updated | `review-code` | claude-review |
+| Rework | Review: changes requested | `rework-pr` | claude-dev |
+| Test | Review: approved | `test` | claude-review |
+| Deploy | Label: `ready-for-deployment` | `devops` | claude-dev |
+| Fix Bug | Issue opened with `bug` label | `fix-bug` | claude-dev |
+
+## Infrastructure Setup
 
 ### Prerequisites
 
-- Node.js 22+
+- k3s cluster with Traefik ingress and cert-manager
-- Coder running with the `cloudflare-worker` template (see `coder/`)
+- Coder instance (e.g. `coder.samson.media`)
-- Gitea with two bot accounts: `claude-dev` and `claude-review`
+- Gitea instance (e.g. `gitea.samson.media`)
+- Docker registry (e.g. `registry.samson.media`)
+- Claude Code Max subscription (for OAuth token)
+
+### Deploy the Orchestrator
+
+1. **Build the workspace image** (pre-installed Node.js, Claude Code, Playwright deps):
+   ```bash
+   docker build -f coder/workspace.Dockerfile -t registry.samson.media/coder-workspace:latest .
+   docker push registry.samson.media/coder-workspace:latest
+   ```
+
+2. **Push the Coder template**:
+   ```bash
+   coder templates push cloudflare-worker --directory coder/cloudflare-worker --yes
+   ```
+   Note the template ID from the Coder dashboard.
+
+3. **Create Gitea bot accounts**:
+   - `claude-dev` — used for most stages (analyse, develop, etc.)
+   - `claude-review` — used for review and test stages
+   - Generate API tokens for each
+
+4. **Generate Claude Code OAuth token** (uses Max subscription instead of API credits):
+   ```bash
+   claude setup-token
+   ```
+   This opens a browser for auth and outputs a long-lived token (`sk-ant-oat01-...`).
+
+5. **Create k8s secrets**:
+   ```bash
+   kubectl create namespace sdlc-orchestrator
+   kubectl create secret generic orchestrator-secrets -n sdlc-orchestrator \
+     --from-literal=CODER_URL=https://coder.samson.media \
+     --from-literal=CODER_TOKEN=<coder-session-token> \
+     --from-literal=CODER_TEMPLATE_ID=<template-id> \
+     --from-literal=GITEA_DEV_TOKEN=<claude-dev-token> \
+     --from-literal=GITEA_REVIEW_TOKEN=<claude-review-token> \
+     --from-literal=CLAUDE_OAUTH_TOKEN=<oauth-token-from-step-4> \
+     --from-literal=CALLBACK_URL=https://sdlc.samson.media \
+     --from-literal=GITEA_URL=https://gitea.samson.media \
+     --from-literal=BOT_DEV_USERNAME=claude-dev \
+     --from-literal=BOT_REVIEW_USERNAME=claude-review
+   ```
+
+6. **Deploy**:
+   ```bash
+   # Tag to trigger CI build
+   git tag -a v1.0.0 -m "Initial release"
+   git push origin main --tags
+
+   # Or deploy manually
+   docker build -t registry.samson.media/sdlc-orchestrator:latest .
+   docker push registry.samson.media/sdlc-orchestrator:latest
+   kubectl apply -f k8s/deployment.yaml
+   ```
 
 ### Environment Variables
 
@@ -43,58 +188,58 @@ Plus a cron job (Monday 9 AM) for weekly maintenance.
 | `CODER_URL` | Yes | Coder API base URL |
 | `CODER_TOKEN` | Yes | Coder session token |
 | `CODER_TEMPLATE_ID` | Yes | Coder workspace template ID |
-| `GITEA_DEV_TOKEN` | Yes | Gitea API token for `claude-dev` |
+| `GITEA_DEV_TOKEN` | Yes | Gitea API token for claude-dev |
-| `GITEA_REVIEW_TOKEN` | Yes | Gitea API token for `claude-review` |
+| `GITEA_REVIEW_TOKEN` | Yes | Gitea API token for claude-review |
-| `ANTHROPIC_API_KEY` | Yes | Anthropic API key for Claude Code |
+| `CLAUDE_OAUTH_TOKEN` | Yes | Claude Code Max OAuth token |
+| `CALLBACK_URL` | Yes | Public URL of this service (e.g. `https://sdlc.samson.media`) |
 | `GITEA_URL` | No | Gitea base URL (default: `https://gitea.samson.media`) |
 | `BOT_DEV_USERNAME` | No | Dev bot username (default: `claude-dev`) |
 | `BOT_REVIEW_USERNAME` | No | Review bot username (default: `claude-review`) |
+| `QUEUE_CONCURRENCY` | No | Max concurrent workspaces (default: 2) |
 | `MAINTENANCE_REPOS` | No | Comma-separated `org/repo=clone_url` for weekly maintenance |
 
-### Local Development
+> **Important**: Do NOT set `ANTHROPIC_API_KEY` in the workspace — it takes priority over the OAuth token and will use pay-per-use credits instead of your Max subscription.
 
-```bash
+## Webhook Endpoints
-npm install
-cp k8s/secret.yaml.example .env  # Edit with real values (use KEY=value format)
-npm run dev
-```
 
-### Deploy to k3s
+| Endpoint | Purpose |
+|----------|---------|
+| `GET /health` | Health check |
+| `GET /queue` | Queue status (pending, running, active workspaces) |
+| `POST /webhook/gitea-issue-triage` | Route new issues to analyse or fix-bug |
+| `POST /webhook/gitea-issue-label` | Stage transitions via labels |
+| `POST /webhook/gitea-issue-comment` | Re-trigger analyst on new/edited comments |
+| `POST /webhook/gitea-pr-review` | Trigger code review on PR open/update |
+| `POST /webhook/gitea-pr-review-rework` | Route review outcomes to rework or test |
+| `POST /webhook/gitea-release` | Handle release workflow |
+| `POST /webhook/task-complete/:name` | Callback from workspaces when done |
 
-```bash
+## How Workspaces Work
-# Build and push image
-docker build -t registry.samson.media/sdlc-orchestrator:latest .
-docker push registry.samson.media/sdlc-orchestrator:latest
 
-# Create secrets
+Workspaces are **ephemeral** (emptyDir, no persistent storage):
-cp k8s/secret.yaml.example k8s/secret.yaml
-# Edit k8s/secret.yaml with real values
-kubectl apply -f k8s/secret.yaml
 
-# Deploy
+1. Orchestrator creates workspace via Coder API with parameters (issue number, task type, tokens, callback URL)
-kubectl apply -f k8s/deployment.yaml
+2. Startup script clones the repo and checks out `develop`
-```
+3. For heavy stages (develop, test, review, rework, fix-bug, devops): installs deps, builds frontend, installs Playwright
+4. For lightweight stages (analyse, architect, release, maintenance): skips build
+5. Reads `.claude/commands/<task_type>.md`, strips YAML frontmatter, substitutes `$ARGUMENTS` with issue number
+6. Runs `claude -p --dangerously-skip-permissions --verbose "<prompt>"`
+7. On completion (success or failure via ERR/EXIT trap): POSTs to callback URL
+8. Orchestrator receives callback, deletes workspace via Coder API
 
-### Configure Gitea Webhooks
+## Resilience
 
-For each project, add these webhooks in **Settings → Webhooks**:
+- **Deduplication**: Tasks keyed by `{taskType}-{repo}-{issue}`. Duplicate webhooks are dropped.
+- **Startup reconciliation**: On boot, queries Coder for running workspaces and re-adopts them into the queue.
+- **409 handling**: If workspace already exists — adopts if running, deletes and retries if stopped/failed.
+- **Callback trap**: Startup script uses `trap` to always fire the callback, even on clone failure or other errors.
+- **TTL safety net**: Coder template has 1-hour auto-stop as a backstop for missed callbacks.
 
-| Event | URL |
+## CI/CD
-|-------|-----|
-| Issues (opened) | `https://sdlc.samson.media/webhook/gitea-issue-triage` |
-| Issues (labeled) | `https://sdlc.samson.media/webhook/gitea-issue-label` |
-| Issue Comments (created) | `https://sdlc.samson.media/webhook/gitea-issue-comment` |
-| Pull Request (opened, synchronized) | `https://sdlc.samson.media/webhook/gitea-pr-review` |
-| Pull Request Review (submitted) | `https://sdlc.samson.media/webhook/gitea-pr-review-rework` |
-| Issues (opened, labeled) — releases | `https://sdlc.samson.media/webhook/gitea-release` |
 
-## Coder Template
+| Workflow | Trigger | Output |
+|----------|---------|--------|
+| `publish.yml` | `v*` tags | `registry.samson.media/sdlc-orchestrator:<version>` + `:latest` |
+| `publish-workspace.yml` | Changes to `coder/workspace.Dockerfile` on main | `registry.samson.media/coder-workspace:latest` |
 
-The `coder/` directory contains the Terraform template for ephemeral Kubernetes workspaces. See `coder/README.md`.
+Fleet GitOps (in `samson-media/devops` repo) watches `registry.samson.media/sdlc-orchestrator:latest` for deployment.
-
-## Migrating from n8n
-
-1. Deploy this service to k3s
-2. Update Gitea webhooks to point to `sdlc.samson.media` instead of `n8n.samson.media`
-3. Verify with a test issue
-4. Decommission n8n

coder/cloudflare-worker/main.tf

@@ -423,7 +423,6 @@ resource "coder_agent" "main" {
   }
 
   env = {
-    ANTHROPIC_API_KEY        = data.coder_parameter.anthropic_api_key.value
     CLAUDE_CODE_OAUTH_TOKEN  = data.coder_parameter.claude_oauth_token.value
     GITHUB_TOKEN      = data.coder_external_auth.github.access_token
     GITEA_TOKEN       = data.coder_parameter.gitea_token.value
@@ -581,11 +580,6 @@ resource "kubernetes_deployment_v1" "workspace" {
             value = coder_agent.main.token
           }
 
-          env {
-            name  = "ANTHROPIC_API_KEY"
-            value = data.coder_parameter.anthropic_api_key.value
-          }
-
           env {
             name  = "CLAUDE_CODE_OAUTH_TOKEN"
             value = data.coder_parameter.claude_oauth_token.value

k8s/redis.yaml

@@ -0,0 +1,69 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: redis-data
+  namespace: sdlc-orchestrator
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  namespace: sdlc-orchestrator
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: redis
+  template:
+    metadata:
+      labels:
+        app: redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:7-alpine
+          args: ["--appendonly", "yes"]
+          ports:
+            - containerPort: 6379
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 256Mi
+          livenessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 30
+          readinessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 3
+            periodSeconds: 10
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: redis-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: sdlc-orchestrator
+spec:
+  selector:
+    app: redis
+  ports:
+    - port: 6379
+      targetPort: 6379

k8s/secret.yaml.example

@@ -15,8 +15,11 @@ stringData:
   GITEA_DEV_TOKEN: "<gitea-claude-dev-token>"
   GITEA_REVIEW_TOKEN: "<gitea-claude-review-token>"
   ANTHROPIC_API_KEY: "<anthropic-api-key>"
+  CLAUDE_OAUTH_TOKEN: "<claude-code-max-oauth-token>"
   GITEA_URL: "https://gitea.samson.media"
   BOT_DEV_USERNAME: "claude-dev"
   BOT_REVIEW_USERNAME: "claude-review"
+  CALLBACK_URL: "https://sdlc.samson.media"
+  REDIS_URL: "redis://redis.sdlc-orchestrator.svc.cluster.local:6379"
   # Comma-separated: org/repo=clone_url
   MAINTENANCE_REPOS: "claude/babble=https://gitea.samson.media/claude/babble.git"

package-lock.json

@@ -9,12 +9,12 @@
       "version": "1.0.0",
       "dependencies": {
         "@hono/node-server": "^1.13.8",
+        "bullmq": "^5.34.8",
         "hono": "^4.7.6",
-        "node-cron": "^3.0.3"
+        "ioredis": "^5.6.1"
       },
       "devDependencies": {
         "@types/node": "^22.15.3",
-        "@types/node-cron": "^3.0.11",
         "eslint": "^9.25.1",
         "prettier": "^3.5.3",
         "tsx": "^4.19.4",
@@ -671,6 +671,90 @@
         "url": "https://github.com/sponsors/nzakas"
       }
     },
+    "node_modules/@ioredis/commands": {
+      "version": "1.5.1",
+      "resolved": "https://registry.npmjs.org/@ioredis/commands/-/commands-1.5.1.tgz",
+      "integrity": "sha512-JH8ZL/ywcJyR9MmJ5BNqZllXNZQqQbnVZOqpPQqE1vHiFgAw4NHbvE0FOduNU8IX9babitBT46571OnPTT0Zcw==",
+      "license": "MIT"
+    },
+    "node_modules/@msgpackr-extract/msgpackr-extract-darwin-arm64": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-darwin-arm64/-/msgpackr-extract-darwin-arm64-3.0.3.tgz",
+      "integrity": "sha512-QZHtlVgbAdy2zAqNA9Gu1UpIuI8Xvsd1v8ic6B2pZmeFnFcMWiPLfWXh7TVw4eGEZ/C9TH281KwhVoeQUKbyjw==",
+      "cpu": [
+        "arm64"
+      ],
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
+    "node_modules/@msgpackr-extract/msgpackr-extract-darwin-x64": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-darwin-x64/-/msgpackr-extract-darwin-x64-3.0.3.tgz",
+      "integrity": "sha512-mdzd3AVzYKuUmiWOQ8GNhl64/IoFGol569zNRdkLReh6LRLHOXxU4U8eq0JwaD8iFHdVGqSy4IjFL4reoWCDFw==",
+      "cpu": [
+        "x64"
+      ],
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
+    "node_modules/@msgpackr-extract/msgpackr-extract-linux-arm": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-linux-arm/-/msgpackr-extract-linux-arm-3.0.3.tgz",
+      "integrity": "sha512-fg0uy/dG/nZEXfYilKoRe7yALaNmHoYeIoJuJ7KJ+YyU2bvY8vPv27f7UKhGRpY6euFYqEVhxCFZgAUNQBM3nw==",
+      "cpu": [
+        "arm"
+      ],
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@msgpackr-extract/msgpackr-extract-linux-arm64": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-linux-arm64/-/msgpackr-extract-linux-arm64-3.0.3.tgz",
+      "integrity": "sha512-YxQL+ax0XqBJDZiKimS2XQaf+2wDGVa1enVRGzEvLLVFeqa5kx2bWbtcSXgsxjQB7nRqqIGFIcLteF/sHeVtQg==",
+      "cpu": [
+        "arm64"
+      ],
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@msgpackr-extract/msgpackr-extract-linux-x64": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-linux-x64/-/msgpackr-extract-linux-x64-3.0.3.tgz",
+      "integrity": "sha512-cvwNfbP07pKUfq1uH+S6KJ7dT9K8WOE4ZiAcsrSes+UY55E/0jLYc+vq+DO7jlmqRb5zAggExKm0H7O/CBaesg==",
+      "cpu": [
+        "x64"
+      ],
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "linux"
+      ]
+    },
+    "node_modules/@msgpackr-extract/msgpackr-extract-win32-x64": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/@msgpackr-extract/msgpackr-extract-win32-x64/-/msgpackr-extract-win32-x64-3.0.3.tgz",
+      "integrity": "sha512-x0fWaQtYp4E6sktbsdAqnehxDgEc/VwM7uLsRCYWaiGu0ykYdZPiS8zCWdnjHwyiumousxfBm4SO31eXqwEZhQ==",
+      "cpu": [
+        "x64"
+      ],
+      "license": "MIT",
+      "optional": true,
+      "os": [
+        "win32"
+      ]
+    },
     "node_modules/@types/estree": {
       "version": "1.0.8",
       "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
@@ -695,13 +779,6 @@
         "undici-types": "~6.21.0"
       }
     },
-    "node_modules/@types/node-cron": {
-      "version": "3.0.11",
-      "resolved": "https://registry.npmjs.org/@types/node-cron/-/node-cron-3.0.11.tgz",
-      "integrity": "sha512-0ikrnug3/IyneSHqCBeslAhlK2aBfYek1fGo4bP4QnZPmiqSGRK+Oy7ZMisLWkesffJvQ1cqAcBnJC+8+nxIAg==",
-      "dev": true,
-      "license": "MIT"
-    },
     "node_modules/acorn": {
       "version": "8.16.0",
       "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.16.0.tgz",
@@ -783,6 +860,34 @@
         "concat-map": "0.0.1"
       }
     },
+    "node_modules/bullmq": {
+      "version": "5.73.4",
+      "resolved": "https://registry.npmjs.org/bullmq/-/bullmq-5.73.4.tgz",
+      "integrity": "sha512-Q+NeFLtdKSD3GDPYSX4pH+Mc9E4OZVKimXwrnZ5WmndNy31COMy4vQV9zfhgfHGSUFrlpsBicfKYbSjx9FbO+A==",
+      "license": "MIT",
+      "dependencies": {
+        "cron-parser": "4.9.0",
+        "ioredis": "5.10.1",
+        "msgpackr": "1.11.5",
+        "node-abort-controller": "3.1.1",
+        "semver": "7.7.4",
+        "tslib": "2.8.1",
+        "uuid": "11.1.0"
+      }
+    },
+    "node_modules/bullmq/node_modules/uuid": {
+      "version": "11.1.0",
+      "resolved": "https://registry.npmjs.org/uuid/-/uuid-11.1.0.tgz",
+      "integrity": "sha512-0/A9rDy9P7cJ+8w1c9WD9V//9Wj15Ce2MPz8Ri6032usz+NfePxx5AcN3bN+r6ZL6jEo066/yNYB3tn4pQEx+A==",
+      "funding": [
+        "https://github.com/sponsors/broofa",
+        "https://github.com/sponsors/ctavan"
+      ],
+      "license": "MIT",
+      "bin": {
+        "uuid": "dist/esm/bin/uuid"
+      }
+    },
     "node_modules/callsites": {
       "version": "3.1.0",
       "resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@@ -810,6 +915,15 @@
         "url": "https://github.com/chalk/chalk?sponsor=1"
       }
     },
+    "node_modules/cluster-key-slot": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",
+      "integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==",
+      "license": "Apache-2.0",
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
     "node_modules/color-convert": {
       "version": "2.0.1",
       "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
@@ -837,6 +951,18 @@
       "dev": true,
       "license": "MIT"
     },
+    "node_modules/cron-parser": {
+      "version": "4.9.0",
+      "resolved": "https://registry.npmjs.org/cron-parser/-/cron-parser-4.9.0.tgz",
+      "integrity": "sha512-p0SaNjrHOnQeR8/VnfGbmg9te2kfyYSQ7Sc/j/6DtPL3JQvKxmjO9TSjNFpujqV3vEYYBvNNvXSxzyksBWAx1Q==",
+      "license": "MIT",
+      "dependencies": {
+        "luxon": "^3.2.1"
+      },
+      "engines": {
+        "node": ">=12.0.0"
+      }
+    },
     "node_modules/cross-spawn": {
       "version": "7.0.6",
       "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
@@ -856,7 +982,6 @@
       "version": "4.4.3",
       "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
       "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
-      "dev": true,
       "license": "MIT",
       "dependencies": {
         "ms": "^2.1.3"
@@ -877,6 +1002,25 @@
       "dev": true,
       "license": "MIT"
     },
+    "node_modules/denque": {
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",
+      "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==",
+      "license": "Apache-2.0",
+      "engines": {
+        "node": ">=0.10"
+      }
+    },
+    "node_modules/detect-libc": {
+      "version": "2.1.2",
+      "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
+      "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
+      "license": "Apache-2.0",
+      "optional": true,
+      "engines": {
+        "node": ">=8"
+      }
+    },
     "node_modules/esbuild": {
       "version": "0.27.7",
       "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
@@ -1268,6 +1412,30 @@
         "node": ">=0.8.19"
       }
     },
+    "node_modules/ioredis": {
+      "version": "5.10.1",
+      "resolved": "https://registry.npmjs.org/ioredis/-/ioredis-5.10.1.tgz",
+      "integrity": "sha512-HuEDBTI70aYdx1v6U97SbNx9F1+svQKBDo30o0b9fw055LMepzpOOd0Ccg9Q6tbqmBSJaMuY0fB7yw9/vjBYCA==",
+      "license": "MIT",
+      "dependencies": {
+        "@ioredis/commands": "1.5.1",
+        "cluster-key-slot": "^1.1.0",
+        "debug": "^4.3.4",
+        "denque": "^2.1.0",
+        "lodash.defaults": "^4.2.0",
+        "lodash.isarguments": "^3.1.0",
+        "redis-errors": "^1.2.0",
+        "redis-parser": "^3.0.0",
+        "standard-as-callback": "^2.1.0"
+      },
+      "engines": {
+        "node": ">=12.22.0"
+      },
+      "funding": {
+        "type": "opencollective",
+        "url": "https://opencollective.com/ioredis"
+      }
+    },
     "node_modules/is-extglob": {
       "version": "2.1.1",
       "resolved": "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz",
@@ -1372,6 +1540,18 @@
         "url": "https://github.com/sponsors/sindresorhus"
       }
     },
+    "node_modules/lodash.defaults": {
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/lodash.defaults/-/lodash.defaults-4.2.0.tgz",
+      "integrity": "sha512-qjxPLHd3r5DnsdGacqOMU6pb/avJzdh9tFX2ymgoZE27BmjXrNy/y4LoaiTeAb+O3gL8AfpJGtqfX/ae2leYYQ==",
+      "license": "MIT"
+    },
+    "node_modules/lodash.isarguments": {
+      "version": "3.1.0",
+      "resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz",
+      "integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==",
+      "license": "MIT"
+    },
     "node_modules/lodash.merge": {
       "version": "4.6.2",
       "resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz",
@@ -1379,6 +1559,15 @@
       "dev": true,
       "license": "MIT"
     },
+    "node_modules/luxon": {
+      "version": "3.7.2",
+      "resolved": "https://registry.npmjs.org/luxon/-/luxon-3.7.2.tgz",
+      "integrity": "sha512-vtEhXh/gNjI9Yg1u4jX/0YVPMvxzHuGgCm6tC5kZyb08yjGWGnqAjGJvcXbqQR2P3MyMEFnRbpcdFS6PBcLqew==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=12"
+      }
+    },
     "node_modules/minimatch": {
       "version": "3.1.5",
       "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz",
@@ -1396,9 +1585,39 @@
       "version": "2.1.3",
       "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
       "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
-      "dev": true,
       "license": "MIT"
     },
+    "node_modules/msgpackr": {
+      "version": "1.11.5",
+      "resolved": "https://registry.npmjs.org/msgpackr/-/msgpackr-1.11.5.tgz",
+      "integrity": "sha512-UjkUHN0yqp9RWKy0Lplhh+wlpdt9oQBYgULZOiFhV3VclSF1JnSQWZ5r9gORQlNYaUKQoR8itv7g7z1xDDuACA==",
+      "license": "MIT",
+      "optionalDependencies": {
+        "msgpackr-extract": "^3.0.2"
+      }
+    },
+    "node_modules/msgpackr-extract": {
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/msgpackr-extract/-/msgpackr-extract-3.0.3.tgz",
+      "integrity": "sha512-P0efT1C9jIdVRefqjzOQ9Xml57zpOXnIuS+csaB4MdZbTdmGDLo8XhzBG1N7aO11gKDDkJvBLULeFTo46wwreA==",
+      "hasInstallScript": true,
+      "license": "MIT",
+      "optional": true,
+      "dependencies": {
+        "node-gyp-build-optional-packages": "5.2.2"
+      },
+      "bin": {
+        "download-msgpackr-prebuilds": "bin/download-prebuilds.js"
+      },
+      "optionalDependencies": {
+        "@msgpackr-extract/msgpackr-extract-darwin-arm64": "3.0.3",
+        "@msgpackr-extract/msgpackr-extract-darwin-x64": "3.0.3",
+        "@msgpackr-extract/msgpackr-extract-linux-arm": "3.0.3",
+        "@msgpackr-extract/msgpackr-extract-linux-arm64": "3.0.3",
+        "@msgpackr-extract/msgpackr-extract-linux-x64": "3.0.3",
+        "@msgpackr-extract/msgpackr-extract-win32-x64": "3.0.3"
+      }
+    },
     "node_modules/natural-compare": {
       "version": "1.4.0",
       "resolved": "https://registry.npmjs.org/natural-compare/-/natural-compare-1.4.0.tgz",
@@ -1406,16 +1625,25 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/node-cron": {
+    "node_modules/node-abort-controller": {
-      "version": "3.0.3",
+      "version": "3.1.1",
-      "resolved": "https://registry.npmjs.org/node-cron/-/node-cron-3.0.3.tgz",
+      "resolved": "https://registry.npmjs.org/node-abort-controller/-/node-abort-controller-3.1.1.tgz",
-      "integrity": "sha512-dOal67//nohNgYWb+nWmg5dkFdIwDm8EpeGYMekPMrngV3637lqnX0lbUcCtgibHTz6SEz7DAIjKvKDFYCnO1A==",
+      "integrity": "sha512-AGK2yQKIjRuqnc6VkX2Xj5d+QW8xZ87pa1UK6yA6ouUyuxfHuMP6umE5QK7UmTeOAymo+Zx1Fxiuw9rVx8taHQ==",
-      "license": "ISC",
+      "license": "MIT"
-      "dependencies": {
-        "uuid": "8.3.2"
     },
-      "engines": {
+    "node_modules/node-gyp-build-optional-packages": {
-        "node": ">=6.0.0"
+      "version": "5.2.2",
+      "resolved": "https://registry.npmjs.org/node-gyp-build-optional-packages/-/node-gyp-build-optional-packages-5.2.2.tgz",
+      "integrity": "sha512-s+w+rBWnpTMwSFbaE0UXsRlg7hU4FjekKU4eyAih5T8nJuNZT1nNsskXpxmeqSK9UzkBl6UgRlnKc8hz8IEqOw==",
+      "license": "MIT",
+      "optional": true,
+      "dependencies": {
+        "detect-libc": "^2.0.1"
+      },
+      "bin": {
+        "node-gyp-build-optional-packages": "bin.js",
+        "node-gyp-build-optional-packages-optional": "optional.js",
+        "node-gyp-build-optional-packages-test": "build-test.js"
       }
     },
     "node_modules/optionator": {
@@ -1537,6 +1765,27 @@
         "node": ">=6"
       }
     },
+    "node_modules/redis-errors": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",
+      "integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=4"
+      }
+    },
+    "node_modules/redis-parser": {
+      "version": "3.0.0",
+      "resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-3.0.0.tgz",
+      "integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==",
+      "license": "MIT",
+      "dependencies": {
+        "redis-errors": "^1.0.0"
+      },
+      "engines": {
+        "node": ">=4"
+      }
+    },
     "node_modules/resolve-from": {
       "version": "4.0.0",
       "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-4.0.0.tgz",
@@ -1557,6 +1806,18 @@
         "url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1"
       }
     },
+    "node_modules/semver": {
+      "version": "7.7.4",
+      "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz",
+      "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==",
+      "license": "ISC",
+      "bin": {
+        "semver": "bin/semver.js"
+      },
+      "engines": {
+        "node": ">=10"
+      }
+    },
     "node_modules/shebang-command": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz",
@@ -1580,6 +1841,12 @@
         "node": ">=8"
       }
     },
+    "node_modules/standard-as-callback": {
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/standard-as-callback/-/standard-as-callback-2.1.0.tgz",
+      "integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==",
+      "license": "MIT"
+    },
     "node_modules/strip-json-comments": {
       "version": "3.1.1",
       "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz",
@@ -1606,6 +1873,12 @@
         "node": ">=8"
       }
     },
+    "node_modules/tslib": {
+      "version": "2.8.1",
+      "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
+      "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
+      "license": "0BSD"
+    },
     "node_modules/tsx": {
       "version": "4.21.0",
       "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz",
@@ -1670,15 +1943,6 @@
         "punycode": "^2.1.0"
       }
     },
-    "node_modules/uuid": {
-      "version": "8.3.2",
-      "resolved": "https://registry.npmjs.org/uuid/-/uuid-8.3.2.tgz",
-      "integrity": "sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg==",
-      "license": "MIT",
-      "bin": {
-        "uuid": "dist/bin/uuid"
-      }
-    },
     "node_modules/which": {
       "version": "2.0.2",
       "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",

package.json

@@ -13,12 +13,12 @@
   },
   "dependencies": {
     "@hono/node-server": "^1.13.8",
+    "bullmq": "^5.34.8",
     "hono": "^4.7.6",
-    "node-cron": "^3.0.3"
+    "ioredis": "^5.6.1"
   },
   "devDependencies": {
     "@types/node": "^22.15.3",
-    "@types/node-cron": "^3.0.11",
     "tsx": "^4.19.4",
     "typescript": "^5.8.3",
     "eslint": "^9.25.1",

src/config.ts

@@ -25,6 +25,14 @@ export interface Config {
   callbackUrl: string;
   /** Optional webhook secret for verifying Gitea signatures */
   webhookSecret?: string;
+  /** Redis connection URL (e.g. redis://redis:6379) */
+  redisUrl: string;
+  /** Max concurrent workspace creations (default: 2) */
+  queueConcurrency: number;
+  /** Max concurrent cleanup jobs (default: 5) */
+  cleanupConcurrency: number;
+  /** Minutes before a workspace is considered stale (default: 120) */
+  staleWorkspaceMinutes: number;
 }
 
 function required(name: string): string {
@@ -50,5 +58,9 @@ export function loadConfig(): Config {
     giteaUrl: process.env.GITEA_URL || "https://gitea.samson.media",
     callbackUrl: required("CALLBACK_URL"),
     webhookSecret: process.env.WEBHOOK_SECRET,
+    redisUrl: required("REDIS_URL"),
+    queueConcurrency: parseInt(process.env.QUEUE_CONCURRENCY || "2", 10),
+    cleanupConcurrency: parseInt(process.env.CLEANUP_CONCURRENCY || "5", 10),
+    staleWorkspaceMinutes: parseInt(process.env.STALE_WORKSPACE_MINUTES || "120", 10),
   };
 }

src/handlers/issue-comment.ts

@@ -43,7 +43,7 @@ export function issueComment(config: Config, queue: TaskQueue) {
       giteaToken: config.giteaDevToken,
     };
 
-    const queued = queue.enqueue(task);
+    const queued = await queue.enqueue(task);
 
     return c.json({ ok: true, queued });
   };

src/handlers/issue-label.ts

@@ -54,7 +54,7 @@ export function issueLabel(config: Config, queue: TaskQueue) {
       giteaToken,
     };
 
-    const queued = queue.enqueue(task, {
+    const queued = await queue.enqueue(task, {
       useReviewAccount: false,
       body: `🤖 SDLC stage transition: **${taskType}** — workspace queued.`,
     });

src/handlers/issue-triage.ts

@@ -31,7 +31,7 @@ export function issueTriage(config: Config, queue: TaskQueue) {
       giteaToken: config.giteaDevToken,
     };
 
-    const queued = queue.enqueue(task, {
+    const queued = await queue.enqueue(task, {
       useReviewAccount: false,
       body: `🤖 Issue received. Starting **${taskType}** stage.`,
     });

src/handlers/maintenance.ts

@@ -25,7 +25,7 @@ export function runMaintenance(config: Config, queue: TaskQueue, repos: Maintena
         giteaToken: config.giteaDevToken,
       };
 
-      queue.enqueue(task);
+      await queue.enqueue(task);
     }
   };
 }

src/handlers/pr-review.ts

@@ -40,7 +40,7 @@ export function prReview(config: Config, queue: TaskQueue, gitea: GiteaClient) {
       giteaToken: config.giteaReviewToken,
     };
 
-    const queued = queue.enqueue(task);
+    const queued = await queue.enqueue(task);
 
     return c.json({ ok: true, queued });
   };

src/handlers/pr-rework.ts

@@ -42,7 +42,7 @@ export function prRework(config: Config, queue: TaskQueue) {
       giteaToken,
     };
 
-    const queued = queue.enqueue(task, {
+    const queued = await queue.enqueue(task, {
       useReviewAccount: false,
       body: `🤖 Review outcome: **${taskType}** — workspace queued.`,
     });

src/handlers/release.ts

@@ -51,7 +51,7 @@ export function release(config: Config, queue: TaskQueue) {
       giteaToken: config.giteaDevToken,
     };
 
-    const queued = queue.enqueue(task, {
+    const queued = await queue.enqueue(task, {
       useReviewAccount: false,
       body: `🤖 Release process started.`,
     });

src/index.ts

@@ -1,7 +1,6 @@
 import { Hono } from "hono";
 import { logger } from "hono/logger";
 import { serve } from "@hono/node-server";
-import cron from "node-cron";
 
 import { loadConfig } from "./config.js";
 import { CoderClient } from "./services/coder.js";
@@ -23,9 +22,7 @@ import { runMaintenance, type MaintenanceRepo } from "./handlers/maintenance.js"
 const config = loadConfig();
 const coderClient = new CoderClient(config);
 const giteaClient = new GiteaClient(config);
-
+const queue = new TaskQueue(config, coderClient, giteaClient);
-const concurrency = parseInt(process.env.QUEUE_CONCURRENCY || "2", 10);
-const queue = new TaskQueue(coderClient, giteaClient, concurrency);
 
 const app = new Hono();
 app.use("*", logger());
@@ -35,7 +32,7 @@ app.use("*", logger());
 // ---------------------------------------------------------------------------
 
 app.get("/health", (c) => c.json({ status: "ok" }));
-app.get("/queue", (c) => c.json(queue.status()));
+app.get("/queue", async (c) => c.json(await queue.status()));
 
 // ---------------------------------------------------------------------------
 // Webhook endpoints — same paths as the n8n webhooks so Gitea config
@@ -61,37 +58,51 @@ app.post("/webhook/task-complete/:name", async (c) => {
 });
 
 // ---------------------------------------------------------------------------
-// Scheduled maintenance cron (Monday 9:00 AM)
+// Maintenance (scheduled via BullMQ repeatable job)
 // ---------------------------------------------------------------------------
 
 const maintenanceRepos = parseMaintenanceRepos();
 if (maintenanceRepos.length > 0) {
   const maintenanceFn = runMaintenance(config, queue, maintenanceRepos);
 
-  cron.schedule("0 9 * * 1", () => {
+  // The maintenance function is called by the queue's repeatable job system.
-    console.log("[cron] running weekly maintenance");
+  // We store the function reference for the maintenance handler to use.
-    maintenanceFn().catch((err) =>
+  console.log(`Maintenance configured for ${maintenanceRepos.length} repo(s) (Monday 9:00 AM via BullMQ)`);
-      console.error("[cron] maintenance failed:", err),
-    );
-  });
-
-  console.log(
-    `Maintenance cron scheduled for ${maintenanceRepos.length} repo(s)`,
-  );
 }
 
 // ---------------------------------------------------------------------------
 // Start
 // ---------------------------------------------------------------------------
 
+async function start() {
+  // Initialize repeatable jobs (stale sweep)
+  await queue.init();
+
   // Reconcile queue state with Coder before accepting traffic
-queue.reconcile().catch((err) =>
+  await queue.reconcile();
-  console.error("[startup] reconcile failed:", err),
-);
 
   serve({ fetch: app.fetch, port: config.port }, (info) => {
-  console.log(`SDLC Orchestrator listening on :${info.port} (concurrency: ${concurrency})`);
+    console.log(`SDLC Orchestrator listening on :${info.port} (concurrency: ${config.queueConcurrency})`);
   });
+}
+
+start().catch((err) => {
+  console.error("[startup] fatal:", err);
+  process.exit(1);
+});
+
+// ---------------------------------------------------------------------------
+// Graceful shutdown
+// ---------------------------------------------------------------------------
+
+const shutdown = async () => {
+  console.log("[shutdown] received signal, shutting down...");
+  await queue.gracefulShutdown();
+  process.exit(0);
+};
+
+process.on("SIGTERM", shutdown);
+process.on("SIGINT", shutdown);
 
 // ---------------------------------------------------------------------------
 // Helpers

src/queue.ts

@@ -1,169 +1,232 @@
+import { Queue, Worker, type Job } from "bullmq";
+import Redis from "ioredis";
 import type { TaskRequest } from "./types.js";
 import type { CoderClient } from "./services/coder.js";
 import type { GiteaClient } from "./services/gitea.js";
+import type { Config } from "./config.js";
+import { dedupKey, workspaceName } from "./workspace-name.js";
 
-interface QueuedTask {
+const ACTIVE_HASH = "sdlc:active-workspaces";
+
+interface CreateJobData {
   task: TaskRequest;
   comment?: { useReviewAccount: boolean; body: string };
 }
 
-interface ActiveWorkspace {
+interface CleanupJobData {
   workspaceId: string;
   workspaceName: string;
-  startedAt: Date;
+}
+
+interface ActiveEntry {
+  workspaceId: string;
+  workspaceName: string;
+  startedAt: number;
 }
 
 /**
- * Task queue with deduplication, concurrency control, and workspace lifecycle.
+ * BullMQ-backed task queue with Redis persistence, retry, and async cleanup.
  *
- * - Tasks are keyed by `{taskType}-{repo}-{issue}` for dedup
+ * Two queues:
- * - If a task with the same key is pending, running, or has an active workspace, new requests are dropped
+ * - `workspace-create`: creates Coder workspaces (concurrency-limited)
- * - Concurrency is configurable (defaults to 2)
+ * - `workspace-cleanup`: stops and deletes workspaces (higher concurrency, with retry)
- * - Tracks active workspaces and stops them on task-complete callback
+ *
+ * Active workspaces (created, awaiting callback) are tracked in a Redis hash
+ * for dedup across restarts.
  */
 export class TaskQueue {
-  private pending = new Map<string, QueuedTask>();
+  private redis: Redis;
-  private running = new Set<string>();
+  private createQueue: Queue<CreateJobData>;
-  private active = new Map<string, ActiveWorkspace>();
+  private cleanupQueue: Queue<CleanupJobData | { sweep: true }>;
-  private concurrency: number;
+  private createWorker: Worker<CreateJobData>;
+  private cleanupWorker: Worker<CleanupJobData | { sweep: true }>;
   private coder: CoderClient;
   private gitea: GiteaClient;
+  private config: Config;
 
-  constructor(coder: CoderClient, gitea: GiteaClient, concurrency = 2) {
+  constructor(config: Config, coder: CoderClient, gitea: GiteaClient) {
+    this.config = config;
     this.coder = coder;
     this.gitea = gitea;
-    this.concurrency = concurrency;
+
+    this.redis = new Redis(config.redisUrl, { maxRetriesPerRequest: null });
+
+    const connection = { connection: this.redis };
+
+    this.createQueue = new Queue("workspace-create", connection);
+    this.cleanupQueue = new Queue("workspace-cleanup", connection);
+
+    this.createWorker = new Worker(
+      "workspace-create",
+      (job) => this.processCreate(job),
+      { ...connection, concurrency: config.queueConcurrency },
+    );
+
+    this.cleanupWorker = new Worker(
+      "workspace-cleanup",
+      (job) => this.processCleanup(job),
+      { ...connection, concurrency: config.cleanupConcurrency },
+    );
+
+    this.createWorker.on("failed", (job, err) => {
+      console.error(`[queue] create job failed: ${job?.id}`, err.message);
+    });
+
+    this.cleanupWorker.on("failed", (job, err) => {
+      console.error(`[queue] cleanup job failed: ${job?.id}`, err.message);
+    });
+  }
+
+  /**
+   * Initialize repeatable jobs (stale sweep).
+   * Call after construction — separated because it's async.
+   */
+  async init(): Promise<void> {
+    await this.cleanupQueue.add(
+      "stale-sweep",
+      { sweep: true } as { sweep: true },
+      {
+        repeat: { every: 10 * 60 * 1000 },
+        jobId: "stale-sweep",
+      },
+    );
+    console.log("[queue] stale sweep scheduled every 10 minutes");
   }
 
   /**
    * Reconcile in-memory state with Coder on startup.
-   * Re-adopts any running workspaces so callbacks and dedup work correctly.
+   * Re-adopts any running workspaces so dedup works correctly.
    */
   async reconcile(): Promise<void> {
     const workspaces = await this.coder.listWorkspaces();
     const runningStatuses = ["starting", "running", "started"];
 
+    let adopted = 0;
     for (const ws of workspaces) {
       if (!runningStatuses.includes(ws.latestBuildStatus)) continue;
 
-      // Only adopt workspaces that match our naming pattern: {taskType}-{repo}-{issue}
+      // Only adopt workspaces that match our naming pattern
       const parts = ws.name.match(/^(.+?)-(.+?)-(\d+)$/);
       if (!parts) continue;
 
-      this.active.set(ws.name, {
+      const existing = await this.redis.hget(ACTIVE_HASH, ws.name);
+      if (existing) continue;
+
+      const entry: ActiveEntry = {
         workspaceId: ws.id,
         workspaceName: ws.name,
-        startedAt: new Date(), // approximate — we don't know the real start time
+        startedAt: Date.now(),
-      });
+      };
+      await this.redis.hset(ACTIVE_HASH, ws.name, JSON.stringify(entry));
+      adopted++;
       console.log(`[queue] reconciled: adopted workspace "${ws.name}" (status: ${ws.latestBuildStatus})`);
     }
 
-    if (this.active.size > 0) {
+    // Clean stale entries from the hash that no longer exist in Coder
-      console.log(`[queue] reconcile complete: ${this.active.size} active workspace(s) adopted`);
+    const activeEntries = await this.redis.hgetall(ACTIVE_HASH);
+    const coderNames = new Set(workspaces.map((w) => w.name));
+    for (const key of Object.keys(activeEntries)) {
+      if (!coderNames.has(key)) {
+        await this.redis.hdel(ACTIVE_HASH, key);
+        console.log(`[queue] reconciled: removed stale entry "${key}" (no matching workspace)`);
       }
     }
 
-  /** Build a dedup key for a task. */
+    if (adopted > 0) {
-  static key(task: TaskRequest): string {
+      console.log(`[queue] reconcile complete: ${adopted} workspace(s) adopted`);
-    return `${task.taskType}-${task.giteaRepo}-${task.issueNumber}`;
     }
-
-  /** Build the workspace name (matches the dedup key). */
-  static workspaceName(task: TaskRequest): string {
-    return `${task.taskType}-${task.giteaRepo}-${task.issueNumber}`;
   }
 
   /**
    * Enqueue a task. Returns true if queued, false if deduplicated (dropped).
    */
-  enqueue(
+  async enqueue(
     task: TaskRequest,
     comment?: { useReviewAccount: boolean; body: string },
-  ): boolean {
+  ): Promise<boolean> {
-    const key = TaskQueue.key(task);
+    const key = dedupKey(task);
 
-    if (this.active.has(key)) {
+    // Check if workspace is already active (created, awaiting callback)
+    const activeEntry = await this.redis.hget(ACTIVE_HASH, key);
+    if (activeEntry) {
       console.log(`[queue] dropped (active workspace): ${key}`);
       return false;
     }
 
-    if (this.running.has(key)) {
+    try {
-      console.log(`[queue] dropped (running): ${key}`);
+      await this.createQueue.add(
-      return false;
+        "create-workspace",
-    }
+        { task, comment },
-
+        {
-    if (this.pending.has(key)) {
+          jobId: `create-${key}`,
-      console.log(`[queue] dropped (pending): ${key}`);
+          attempts: 3,
-      return false;
+          backoff: { type: "exponential", delay: 5000 },
-    }
+        },
-
-    this.pending.set(key, { task, comment });
-    console.log(
-      `[queue] enqueued: ${key} (pending: ${this.pending.size}, running: ${this.running.size}/${this.concurrency}, active: ${this.active.size})`,
       );
+    } catch (err: unknown) {
+      // BullMQ throws when a job with the same ID already exists
+      if (err instanceof Error && err.message.includes("duplicated")) {
+        console.log(`[queue] dropped (already queued): ${key}`);
+        return false;
+      }
+      throw err;
+    }
 
-    this.drain();
+    console.log(`[queue] enqueued: ${key}`);
     return true;
   }
 
   /**
    * Handle task-complete callback from a workspace.
-   * Stops the workspace via Coder API and removes it from active tracking.
+   * Enqueues a cleanup job instead of blocking.
    */
   async taskComplete(workspaceName: string): Promise<boolean> {
-    const entry = this.active.get(workspaceName);
+    const raw = await this.redis.hget(ACTIVE_HASH, workspaceName);
-    if (!entry) {
+    if (!raw) {
       console.log(`[queue] task-complete for unknown workspace: ${workspaceName}`);
       return false;
     }
 
-    const elapsed = Math.round(
+    const entry: ActiveEntry = JSON.parse(raw);
-      (Date.now() - entry.startedAt.getTime()) / 1000,
+    const elapsed = Math.round((Date.now() - entry.startedAt) / 1000);
+    console.log(`[queue] task complete: ${workspaceName} (ran for ${elapsed}s) — queueing cleanup`);
+
+    // Remove from active immediately so new tasks for this key can be queued
+    await this.redis.hdel(ACTIVE_HASH, workspaceName);
+
+    await this.cleanupQueue.add(
+      "cleanup-workspace",
+      { workspaceId: entry.workspaceId, workspaceName: entry.workspaceName },
+      {
+        jobId: `cleanup-${workspaceName}`,
+        attempts: 5,
+        backoff: { type: "exponential", delay: 10000 },
+      },
     );
-    console.log(
-      `[queue] task complete: ${workspaceName} (ran for ${elapsed}s) — stopping workspace`,
-    );
-
-    this.active.delete(workspaceName);
-
-    try {
-      await this.coder.deleteWorkspace(entry.workspaceId);
-      console.log(`[queue] workspace deleted: ${workspaceName}`);
-    } catch (err) {
-      console.error(`[queue] failed to delete workspace ${workspaceName}:`, err);
-    }
-
-    // Drain in case pending tasks were waiting for capacity
-    this.drain();
 
     return true;
   }
 
-  /** Process queued tasks up to the concurrency limit. */
+  // ---------------------------------------------------------------------------
-  private drain(): void {
+  // Workers
-    const totalInFlight = this.running.size + this.active.size;
+  // ---------------------------------------------------------------------------
-    while (totalInFlight + this.pending.size > 0 && this.running.size + this.active.size < this.concurrency && this.pending.size > 0) {
-      const [key, entry] = this.pending.entries().next().value!;
-      this.pending.delete(key);
-      this.running.add(key);
-      this.process(key, entry);
-    }
-  }
 
-  private async process(key: string, entry: QueuedTask): Promise<void> {
+  private async processCreate(job: Job<CreateJobData>): Promise<void> {
-    const { task, comment } = entry;
+    const { task, comment } = job.data;
+    const key = dedupKey(task);
+
+    console.log(`[queue] processing create: ${key} (attempt ${job.attemptsMade + 1})`);
 
-    try {
-      console.log(`[queue] processing: ${key}`);
     const workspace = await this.coder.createWorkspace(task);
     console.log(`[queue] workspace created: ${workspace.name} (id: ${workspace.id})`);
 
-      // Move from running → active (workspace is now alive, waiting for callback)
+    // Track as active
-      this.active.set(key, {
+    const entry: ActiveEntry = {
       workspaceId: workspace.id,
       workspaceName: workspace.name,
-        startedAt: new Date(),
+      startedAt: Date.now(),
-      });
+    };
+    await this.redis.hset(ACTIVE_HASH, key, JSON.stringify(entry));
 
     if (comment) {
       await this.gitea.commentOnIssue(
@@ -174,34 +237,135 @@ export class TaskQueue {
         comment.useReviewAccount,
       );
     }
+  }
+
+  private async processCleanup(
+    job: Job<CleanupJobData | { sweep: true }>,
+  ): Promise<void> {
+    if ("sweep" in job.data) {
+      await this.staleSweep();
+      return;
+    }
+
+    const { workspaceId, workspaceName: wsName } = job.data as CleanupJobData;
+    console.log(`[queue] cleanup: stopping ${wsName} (attempt ${job.attemptsMade + 1})`);
+
+    await this.coder.stopWorkspace(workspaceId);
+
+    // Poll until stopped (max 5 minutes)
+    const maxPolls = 30;
+    const pollInterval = 10_000;
+    const doneStatuses = ["stopped", "failed", "canceled", "deleted"];
+
+    for (let i = 0; i < maxPolls; i++) {
+      await new Promise((r) => setTimeout(r, pollInterval));
+      const ws = await this.coder.findWorkspaceByName(wsName);
+      if (!ws || doneStatuses.includes(ws.latestBuildStatus)) {
+        break;
+      }
+      console.log(`[queue] cleanup: ${wsName} still ${ws.latestBuildStatus}, polling...`);
+    }
+
+    await this.coder.deleteWorkspace(workspaceId);
+    console.log(`[queue] workspace deleted: ${wsName}`);
+  }
+
+  private async staleSweep(): Promise<void> {
+    console.log("[queue] running stale sweep");
+    const workspaces = await this.coder.listWorkspaces();
+    const now = Date.now();
+    const staleMs = this.config.staleWorkspaceMinutes * 60 * 1000;
+    const doneStatuses = ["stopped", "failed", "canceled", "deleted"];
+
+    for (const ws of workspaces) {
+      // Only manage workspaces that match our naming pattern
+      const parts = ws.name.match(/^(.+?)-(.+?)-(\d+)$/);
+      if (!parts) continue;
+
+      if (doneStatuses.includes(ws.latestBuildStatus)) {
+        // Stopped/failed workspace — clean up
+        console.log(`[queue] stale sweep: deleting ${ws.latestBuildStatus} workspace "${ws.name}"`);
+        try {
+          await this.coder.deleteWorkspace(ws.id);
+          await this.redis.hdel(ACTIVE_HASH, ws.name);
         } catch (err) {
-      console.error(`[queue] failed: ${key}`, err);
+          console.error(`[queue] stale sweep: failed to delete "${ws.name}":`, err);
-    } finally {
+        }
-      this.running.delete(key);
+        continue;
-      // Don't drain here — active workspaces count toward concurrency
+      }
+
+      // Check if running workspace is stale based on our tracking
+      const raw = await this.redis.hget(ACTIVE_HASH, ws.name);
+      if (raw) {
+        const entry: ActiveEntry = JSON.parse(raw);
+        if (now - entry.startedAt > staleMs) {
+          console.log(`[queue] stale sweep: workspace "${ws.name}" exceeded ${this.config.staleWorkspaceMinutes}m — stopping`);
+          try {
+            await this.coder.stopWorkspace(ws.id);
+            await this.redis.hdel(ACTIVE_HASH, ws.name);
+            // Enqueue cleanup to handle the stop→delete lifecycle
+            await this.cleanupQueue.add(
+              "cleanup-workspace",
+              { workspaceId: ws.id, workspaceName: ws.name },
+              {
+                jobId: `cleanup-stale-${ws.name}-${Date.now()}`,
+                attempts: 5,
+                backoff: { type: "exponential", delay: 10000 },
+              },
+            );
+          } catch (err) {
+            console.error(`[queue] stale sweep: failed to stop "${ws.name}":`, err);
+          }
+        }
+      }
     }
   }
 
-  /** Current queue status for health/debug endpoints. */
+  // ---------------------------------------------------------------------------
-  status(): {
+  // Status & lifecycle
-    pending: string[];
+  // ---------------------------------------------------------------------------
-    running: string[];
+
-    active: Record<string, { workspaceId: string; elapsed: number }>;
+  async status(): Promise<{
+    pending: number;
+    active: number;
+    activeWorkspaces: Record<string, { workspaceId: string; elapsed: number }>;
+    failed: number;
     concurrency: number;
-  } {
+  }> {
-    const activeMap: Record<string, { workspaceId: string; elapsed: number }> = {};
+    const [waiting, activeJobs, failed] = await Promise.all([
-    for (const [key, entry] of this.active) {
+      this.createQueue.getWaitingCount(),
-      activeMap[key] = {
+      this.createQueue.getActiveCount(),
+      this.createQueue.getFailedCount(),
+    ]);
+
+    const activeEntries = await this.redis.hgetall(ACTIVE_HASH);
+    const activeWorkspaces: Record<string, { workspaceId: string; elapsed: number }> = {};
+    const now = Date.now();
+
+    for (const [key, raw] of Object.entries(activeEntries)) {
+      const entry: ActiveEntry = JSON.parse(raw);
+      activeWorkspaces[key] = {
         workspaceId: entry.workspaceId,
-        elapsed: Math.round((Date.now() - entry.startedAt.getTime()) / 1000),
+        elapsed: Math.round((now - entry.startedAt) / 1000),
       };
     }
 
     return {
-      pending: [...this.pending.keys()],
+      pending: waiting + activeJobs,
-      running: [...this.running.keys()],
+      active: Object.keys(activeWorkspaces).length,
-      active: activeMap,
+      activeWorkspaces,
-      concurrency: this.concurrency,
+      failed,
+      concurrency: this.config.queueConcurrency,
     };
   }
+
+  async gracefulShutdown(): Promise<void> {
+    console.log("[queue] shutting down workers...");
+    await Promise.all([
+      this.createWorker.close(),
+      this.cleanupWorker.close(),
+    ]);
+    this.redis.disconnect();
+    console.log("[queue] shutdown complete");
+  }
 }

src/services/coder.ts

@@ -1,6 +1,6 @@
 import type { Config } from "../config.js";
 import type { TaskRequest } from "../types.js";
-import { TaskQueue } from "../queue.js";
+import { workspaceName as buildWorkspaceName } from "../workspace-name.js";
 
 export class CoderClient {
   private baseUrl: string;
@@ -20,7 +20,7 @@ export class CoderClient {
   }
 
   async createWorkspace(task: TaskRequest): Promise<{ id: string; name: string }> {
-    const name = TaskQueue.workspaceName(task);
+    const name = buildWorkspaceName(task);
 
     const body = {
       name,

src/workspace-name.ts

@@ -0,0 +1,11 @@
+import type { TaskRequest } from "./types.js";
+
+/** Build a dedup key for a task. */
+export function dedupKey(task: TaskRequest): string {
+  return `${task.taskType}-${task.giteaRepo}-${task.issueNumber}`;
+}
+
+/** Build the workspace name (matches the dedup key). */
+export function workspaceName(task: TaskRequest): string {
+  return `${task.taskType}-${task.giteaRepo}-${task.issueNumber}`;
+}