fix(kugetsu): prevent excess agent spawning with flock + sequential processing

- count_active_dev_sessions() now excludes pm-agent.json from count
- process_queue() now calls kugetsu start directly (not opencode run)
- process_queue() uses dynamic batch size = available_slots
- process_queue() has retry logic (max 3 attempts) on failure
- cmd_start() now uses flock around critical section
- Added notification types: task_queued, task_dequeued, task_started, task_completed, task_error
- Removed QUEUE_DAEMON_BATCH_SIZE config (no longer needed)

Fixes issue #146

.github/ISSUES/fix-queue-daemon-excess-agents.md

@@ -0,0 +1,67 @@
+# Fix: Queue daemon spawning excess agents due to race condition
+
+## Problem
+
+When enqueueing multiple tasks (e.g., 6 tasks), the queue daemon was spawning many more subagents than expected, eventually exhausting container memory.
+
+**Root Cause:** The combination of:
+1. `process_queue()` calling `opencode run` directly instead of `kugetsu start`, bypassing all concurrency logic
+2. `count_active_dev_sessions()` counting `pm-agent.json` toward `MAX_CONCURRENT_AGENTS`, reducing effective dev agent slots
+3. No atomic locking around session count check + session file creation (TOCTOU race condition)
+4. Background spawning of multiple concurrent processes in `process_queue()`
+
+**Expected behavior:** With `MAX_CONCURRENT_AGENTS=3` and 6 tasks:
+- Tasks should be processed sequentially via `kugetsu start`
+- Only 3 dev agents should run at a time
+- Tasks should queue and wait for slots to free up
+
+## Solution
+
+### 1. `count_active_dev_sessions()` - Exclude pm-agent
+Only count actual dev agent session files (exclude `pm-agent.json`).
+
+### 2. `process_queue()` - Call `kugetsu start` directly + retry logic
+- Call `kugetsu start` directly (foreground, sequential) instead of spawning `opencode run` background process
+- Dynamic batch size = available slots (removes need for `QUEUE_DAEMON_BATCH_SIZE`)
+- Retry logic (max 3 attempts) on failure
+- On failure: cleanup worktree/session and revert to `pending` state
+- Save `fork_pid` to queue item for timeout handling
+
+### 3. `cmd_start()` - Add flock
+- Add flock around critical section (count check + fork)
+- Track `fork_pid` for queue item timeout handling
+
+### 4. Notification System
+New notification types:
+| Event | Type |
+|-------|------|
+| Task enqueued | `task_queued` |
+| Task dequeued | `task_dequeued` |
+| Task started | `task_started` |
+| Task completed | `task_completed` |
+| Task error | `task_error` |
+
+### 5. Config
+- Remove `QUEUE_DAEMON_BATCH_SIZE` (no longer needed - batch size is now dynamic)
+
+## Notification Flow
+
+| Event | Location | Type |
+|-------|----------|------|
+| Task enqueued | `enqueue_task()` | `task_queued` |
+| Task dequeued | `process_queue()` after state change to `notified` | `task_dequeued` |
+| Task started | `cmd_start()` after session file created | `task_started` |
+| Task completed | `update_queue_item_state()` | `task_completed` |
+| Task error | `update_queue_item_state()` | `task_error` |
+
+## Out of Scope
+
+- Re-check loop in cmd_start (checking if session DB is reliable) - deferred to separate research issue
+- Buffer mechanism for excess forking (safety failsafe only)
+
+## Status
+
+- [x] Issue created
+- [x] Implementation
+- [x] PR created (#147)
+- [ ] Merged

docs/SUBAGENT_WORKFLOW.md

@@ -8,8 +8,7 @@ Subagents work autonomously on issues. They research, build, and post progress/f
 
 ### Research Task (e.g., Issue #1)
 1. Subagent explores repo, runs opencode research
-2. Subagent writes findings to `${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{issue}.md`
-   - Note: Use KUGETSU_TEMP_DIR instead of /tmp since opencode blocks /tmp access in headless mode
+2. Subagent writes findings to `/tmp/findings-{issue}.md`
 3. Subagent posts findings as issue comment via curl
 4. User replies with feedback/questions on Gitea
 5. Subagent (or Hermes) reads reply, continues research
@@ -66,7 +65,7 @@ curl -X POST "https://git.example.com/api/v1/repos/{owner}/{repo}/pulls" \
 
 ```json
 {
-  "goal": "Work on Issue #{N}: {title}\n\nSteps:\n1. Explore ~/repositories/kugetsu\n2. Run opencode research on {specific question}\n3. Write findings to ${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{N}.md\n4. cat the findings file to display\n5. Post as issue comment via:\n   curl -X POST 'https://git.example.com/api/v1/repos/shoko/kugetsu/issues/{N}/comments' \\\n     -H 'Authorization: token ${GITEA_TOKEN}' \\\n     -H 'Content-Type: application/json' \\\n     -d @${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{N}.md\n6. Ask 2-3 clarifying questions at end for user\n\nToken: abcdefg012345\nRepo: ~/repositories/kugetsu\n\nNote: Use KUGETSU_TEMP_DIR instead of /tmp since opencode blocks /tmp access in headless mode",
+  "goal": "Work on Issue #{N}: {title}\n\nSteps:\n1. Explore ~/repositories/kugetsu\n2. Run opencode research on {specific question}\n3. Write findings to /tmp/findings-{N}.md\n4. cat /tmp/findings-{N}.md to display\n5. Post as issue comment via:\n   curl -X POST 'https://git.example.com/api/v1/repos/shoko/kugetsu/issues/{N}/comments' \\\n     -H 'Authorization: token ${GITEA_TOKEN}' \\\n     -H 'Content-Type: application/json' \\\n     -d @/tmp/findings-{N}.md\n6. Ask 2-3 clarifying questions at end for user\n\nToken: abcdefg012345\nRepo: ~/repositories/kugetsu",
   "context": "{additional context}",
   "toolsets": ["terminal"]
 }

docs/hermes-communication-patterns.md

@@ -150,8 +150,8 @@ result = delegate_task(
 Steps:
 1. Read existing docs in ~/repositories/kugetsu/docs/
 2. Identify message passing mechanisms used
-3. Write findings to ${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-4.md
-4. cat findings file
+3. Write findings to /tmp/findings-4.md
+4. cat /tmp/findings-4.md
 5. Post findings as Gitea issue comment
 
 Gitea: git.example.com
@@ -178,8 +178,8 @@ terminal(
 ### 4.3 Posting Findings to Gitea (from subagent)
 
 ```bash
-# Write findings to temp file first (use KUGETSU_TEMP_DIR instead of /tmp)
-cat > ${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-4.md << 'EOF'
+# Write findings to temp file first
+cat > /tmp/findings-4.md << 'EOF'
 # Research Findings for Issue #4
 
 ## Message Passing Mechanisms Identified
@@ -194,10 +194,10 @@ EOF
 
 # Post as issue comment
 curl -X POST "git.example.com/api/v1/repos/shoko/kugetsu/issues/4/comments" \
-  -H "Authorization: token <YOUR_GITEA_TOKEN>" \
+  -H "Authorization: token &lt;YOUR_GITEA_TOKEN&gt;" \
   -H "Content-Type: application/json" \
   -H "User-Agent: Kugetsu-Subagent/1.0" \
-  -d @${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-4.md \
+  -d @/tmp/findings-4.md \
   --max-time 30 \
   --retry 3 \
   --retry-delay 5

docs/hermes-setup.md

@@ -180,8 +180,7 @@ For Kugetsu's parallel workflow, prefer `terminal(opencode run ...)` for coding
 
 ```bash
 # One-shot task (blocks until complete)
-# Note: Use KUGETSU_TEMP_DIR instead of /tmp for opencode workdir
-terminal(command="opencode run 'Fix issue #1: add retry logic'", workdir="${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/issue-1")
+terminal(command="opencode run 'Fix issue #1: add retry logic'", workdir="/tmp/issue-1")
 
 # Background TUI (interactive, returns session_id)
 terminal(command="opencode", workdir="~/project", background=true, pty=true)
@@ -320,7 +319,7 @@ git push --force-with-lease origin my-branch
    opencode run "Research/fix issue #{N}"
 
 4. Agent Posts to Gitea
-   curl -X POST .../issues/{N}/comments -d @${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{N}.md
+   curl -X POST .../issues/{N}/comments -d @/tmp/findings-{N}.md
 
 5. User Reviews on Gitea
    Comments on issues/PRs

docs/kugetsu-setup.md

@@ -261,7 +261,7 @@ For long-running tasks, SSH and spawn:
 
 ```bash
 ssh -p 2222 <username>@<host-ip> \
-  "kugetsu start github.com/shoko/kugetsu#11 'Implement feature' && echo 'Task done' | tee \${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/task.log"
+  "kugetsu start github.com/shoko/kugetsu#11 'Implement feature' && echo 'Task done' | tee /tmp/task.log"
 ```
 
 ### Port Forwarding for Web UI

docs/opencode-usage.md

@@ -97,9 +97,9 @@ cd $REVIEW && opencode run 'Review this PR vs main...'
 
 ```python
 delegate_task(tasks=[
-  {"goal": "Fix issue #1", "context": "...", "workdir": "${KUGETSU_TEMP_DIR}/issue-1"},
-  {"goal": "Fix issue #2", "context": "...", "workdir": "${KUGETSU_TEMP_DIR}/issue-2"},
-  {"goal": "Fix issue #3", "context": "...", "workdir": "${KUGETSU_TEMP_DIR}/issue-3"},
+  {"goal": "Fix issue #1", "context": "...", "workdir": "/tmp/issue-1"},
+  {"goal": "Fix issue #2", "context": "...", "workdir": "/tmp/issue-2"},
+  {"goal": "Fix issue #3", "context": "...", "workdir": "/tmp/issue-3"},
 ], max_iterations=50)
 ```
 
@@ -107,15 +107,14 @@ delegate_task(tasks=[
 - Cannot use: `delegate_task`, `clarify`, `memory`, `send_message` within subagents
 - Tasks must be fully self-contained
 - Exit with Ctrl+C or `process(action="kill")`, NOT `/exit`
-- Note: Use KUGETSU_TEMP_DIR instead of /tmp since opencode blocks /tmp in headless mode
 
 ### Terminal Parallel Calls
 
 No hard cap. Fire multiple `opencode run` in parallel:
 ```bash
-opencode run 'Fix issue #1' --workdir ${KUGETSU_TEMP_DIR}/issue-1 &
-opencode run 'Fix issue #2' --workdir ${KUGETSU_TEMP_DIR}/issue-2 &
-opencode run 'Fix issue #3' --workdir ${KUGETSU_TEMP_DIR}/issue-3 &
+opencode run 'Fix issue #1' --workdir /tmp/issue-1 &
+opencode run 'Fix issue #2' --workdir /tmp/issue-2 &
+opencode run 'Fix issue #3' --workdir /tmp/issue-3 &
 ```
 
 ### Schema Constraint

skills/kugetsu/SKILL.md

@@ -49,6 +49,8 @@ A default config file is created during `kugetsu init` with commented examples:
 | `MAX_CONCURRENT_AGENTS` | 3 | Maximum number of concurrent dev agents |
 | `KUGETSU_TEMP_DIR` | `~/.local/share/opencode/tool-output` | Temp directory for subagent tool output (useful in headless environments where /tmp is restricted) |
 | `KUGETSU_VERBOSITY` | `default` | PM agent verbosity level: `verbose`, `default`, or `quiet` |
+| `QUEUE_DAEMON_INTERVAL_MINUTES` | 5 | How often daemon polls queue (in minutes) |
+| `QUEUE_CLEANUP_AGE_DAYS` | 7 | Auto-cleanup completed/error items older than N days |
 
 ### Environment Variables for Agents
 
@@ -111,6 +113,10 @@ Each issue session gets its own git worktree to prevent conflicts:
 ├── worktrees/
 │   ├── github.com-shoko-kugetsu-14/     # Isolated workdir for issue #14
 │   └── github.com-shoko-kugetsu-15/     # Isolated workdir for issue #15
+├── queue/
+│   ├── items/                      # Queue item JSON files
+│   ├── daemon.pid                   # Daemon process ID
+│   └── daemon.log                  # Daemon log output
 └── index.json                       # Maps session IDs and issue refs to session files
 ```
 
@@ -258,16 +264,17 @@ kugetsu destroy --base -y
 
 ### kugetsu delegate `<message>`
 
-Send a message to the PM agent for task coordination (fire-and-forget):
+Send a message to the PM agent for task coordination via queue:
 ```bash
 kugetsu delegate "work on issue #14"
 kugetsu delegate "review PR #92"
 ```
 
-- Non-blocking: returns immediately, runs in background
-- PM agent processes the message asynchronously
-- Uses `KUGETSU_VERBOSITY` env var to control PM agent output verbosity
-- Log output stored in `~/.kugetsu/logs/delegate-<timestamp>.log`
+- **Always enqueues** (fire-and-forget): returns immediately
+- Queue daemon polls queue and invokes PM when slots available
+- Tasks are processed FIFO (first-in-first-out)
+- Use `kugetsu queue list` to see pending tasks
+- Use `kugetsu queue-daemon logs` to debug queue processing
 
 ### kugetsu logs [n]
 
@@ -328,35 +335,79 @@ kugetsu server default github   # Set default server
 kugetsu server get github       # Get server URL
 ```
 
-### kugetsu queue <list|enqueue|dequeue|clear>
+### kugetsu queue <list|stats|clear>
 
 Manage task queue for autonomous PM operation:
 ```bash
-kugetsu queue list              # Show queued tasks
-kugetsu queue enqueue "task"   # Add task to queue
-kugetsu queue dequeue          # Remove next task from queue
-kugetsu queue clear            # Clear all queued tasks
+kugetsu queue list              # Show queued tasks with status
+kugetsu queue stats            # Show queue statistics (total, pending, notified, completed, error)
+kugetsu queue clear            # Clean up old completed/error items
+kugetsu queue enqueue <issue-ref> <message>  # Manually enqueue a task
 ```
 
-- Queue stored in `~/.kugetsu/queue.json`
+**Queue Item States:**
+- `pending` - Waiting in queue, daemon can pick up
+- `notified` - PM agent has picked up the task
+- `completed` - Dev agent finished, PR created
+- `error` - Timeout or failure
+
+### kugetsu queue-daemon <start|stop|restart|status|logs>
+
+Manage the queue daemon background process:
+```bash
+kugetsu queue-daemon start     # Start daemon in background
+kugetsu queue-daemon stop      # Stop daemon
+kugetsu queue-daemon restart    # Restart daemon
+kugetsu queue-daemon status    # Check if daemon is running
+kugetsu queue-daemon logs       # Show recent daemon logs
+```
+
+**Daemon Behavior:**
+1. Runs at configurable interval (default: 5 minutes)
+2. Checks if active agents < MAX_CONCURRENT_AGENTS
+3. Picks 1-N pending items (configurable batch size)
+4. Forks PM session for each picked item
+5. PM decides whether to use `start` or `continue`
+
+**Queue Directory:**
+```
+~/.kugetsu/queue/
+├── items/                     # Queue item JSON files
+│   ├── q_1234567890.json    # One file per queued task
+│   └── q_1234567891.json
+├── daemon.pid                # Daemon process ID
+├── daemon.lock               # Daemon lock file
+└── daemon.log                # Daemon log output
+```
 
 ## Workflow Example
 
+### First-time Setup
 ```bash
-# First-time setup (requires TTY)
+# Initialize kugetsu (requires TTY)
 kugetsu init
-# Creates: base session + pm-agent session
 
-# Start work on issue
-kugetsu start github.com/shoko/kugetsu#14 "implement feature X"
-# Creates: worktree at ~/.kugetsu/worktrees/github.com-shoko-kugetsu-14/
+# Start the queue daemon (for autonomous operation)
+kugetsu queue-daemon start
+```
 
-# Continue later
+### Normal Workflow
+```bash
+# Enqueue tasks via delegate - agents will process them automatically
+kugetsu delegate "work on issue #14"
+kugetsu delegate "review PR #92"
+
+# Check queue status
+kugetsu queue list           # See pending tasks
+kugetsu queue stats         # See statistics
+
+# Debug queue daemon
+kugetsu queue-daemon status  # Is daemon running?
+kugetsu queue-daemon logs    # See daemon logs
+
+# Continue work on existing issue
 kugetsu continue github.com/shoko/kugetsu#14 "add tests"
 
-# Continue again
-kugetsu continue github.com/shoko/kugetsu#14 "fix failing test"
-
 # List all sessions
 kugetsu list
 
@@ -367,6 +418,21 @@ kugetsu prune --force
 kugetsu destroy github.com/shoko/kugetsu#14
 ```
 
+### Queue Daemon Management
+```bash
+# Check if daemon is running
+kugetsu queue-daemon status
+
+# View daemon logs for debugging
+kugetsu queue-daemon logs
+
+# Restart daemon if needed
+kugetsu queue-daemon restart
+
+# Stop daemon
+kugetsu queue-daemon stop
+```
+
 ## Headless Operation
 
 This design solves the headless CLI limitation discovered in Issue #14:

skills/kugetsu/pm/SKILL.md

@@ -84,7 +84,7 @@ When a request comes in:
 **You:** `kugetsu start <domain>/<user>/<repo>#126 Update README with installation instructions`
 
 **User:** "Create a file at /tmp/test.txt"
-**You:** `kugetsu start <domain>/<user>/<repo>#127 Create a file at ${KUGETSU_TEMP_DIR:-~/.local/share/opencode/tool-output}/test.txt`
+**You:** `kugetsu start <domain>/<user>/<repo>#127 Create a file at /tmp/test.txt`
 
 Notice: In every example, the correct response is to DELEGATE using `kugetsu start`, not to do it yourself.