Compare commits
3 Commits
v0.2.1
...
efd25bac59
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
efd25bac59 | ||
|
|
84352799b2 | ||
|
|
8f3c14b2f6 |
67
.github/ISSUES/fix-queue-daemon-excess-agents.md
vendored
67
.github/ISSUES/fix-queue-daemon-excess-agents.md
vendored
@@ -1,67 +0,0 @@
|
||||
# Fix: Queue daemon spawning excess agents due to race condition
|
||||
|
||||
## Problem
|
||||
|
||||
When enqueueing multiple tasks (e.g., 6 tasks), the queue daemon was spawning many more subagents than expected, eventually exhausting container memory.
|
||||
|
||||
**Root Cause:** The combination of:
|
||||
1. `process_queue()` calling `opencode run` directly instead of `kugetsu start`, bypassing all concurrency logic
|
||||
2. `count_active_dev_sessions()` counting `pm-agent.json` toward `MAX_CONCURRENT_AGENTS`, reducing effective dev agent slots
|
||||
3. No atomic locking around session count check + session file creation (TOCTOU race condition)
|
||||
4. Background spawning of multiple concurrent processes in `process_queue()`
|
||||
|
||||
**Expected behavior:** With `MAX_CONCURRENT_AGENTS=3` and 6 tasks:
|
||||
- Tasks should be processed sequentially via `kugetsu start`
|
||||
- Only 3 dev agents should run at a time
|
||||
- Tasks should queue and wait for slots to free up
|
||||
|
||||
## Solution
|
||||
|
||||
### 1. `count_active_dev_sessions()` - Exclude pm-agent
|
||||
Only count actual dev agent session files (exclude `pm-agent.json`).
|
||||
|
||||
### 2. `process_queue()` - Call `kugetsu start` directly + retry logic
|
||||
- Call `kugetsu start` directly (foreground, sequential) instead of spawning `opencode run` background process
|
||||
- Dynamic batch size = available slots (removes need for `QUEUE_DAEMON_BATCH_SIZE`)
|
||||
- Retry logic (max 3 attempts) on failure
|
||||
- On failure: cleanup worktree/session and revert to `pending` state
|
||||
- Save `fork_pid` to queue item for timeout handling
|
||||
|
||||
### 3. `cmd_start()` - Add flock
|
||||
- Add flock around critical section (count check + fork)
|
||||
- Track `fork_pid` for queue item timeout handling
|
||||
|
||||
### 4. Notification System
|
||||
New notification types:
|
||||
| Event | Type |
|
||||
|-------|------|
|
||||
| Task enqueued | `task_queued` |
|
||||
| Task dequeued | `task_dequeued` |
|
||||
| Task started | `task_started` |
|
||||
| Task completed | `task_completed` |
|
||||
| Task error | `task_error` |
|
||||
|
||||
### 5. Config
|
||||
- Remove `QUEUE_DAEMON_BATCH_SIZE` (no longer needed - batch size is now dynamic)
|
||||
|
||||
## Notification Flow
|
||||
|
||||
| Event | Location | Type |
|
||||
|-------|----------|------|
|
||||
| Task enqueued | `enqueue_task()` | `task_queued` |
|
||||
| Task dequeued | `process_queue()` after state change to `notified` | `task_dequeued` |
|
||||
| Task started | `cmd_start()` after session file created | `task_started` |
|
||||
| Task completed | `update_queue_item_state()` | `task_completed` |
|
||||
| Task error | `update_queue_item_state()` | `task_error` |
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Re-check loop in cmd_start (checking if session DB is reliable) - deferred to separate research issue
|
||||
- Buffer mechanism for excess forking (safety failsafe only)
|
||||
|
||||
## Status
|
||||
|
||||
- [x] Issue created
|
||||
- [x] Implementation
|
||||
- [x] PR created (#147)
|
||||
- [ ] Merged
|
||||
@@ -8,7 +8,8 @@ Subagents work autonomously on issues. They research, build, and post progress/f
|
||||
|
||||
### Research Task (e.g., Issue #1)
|
||||
1. Subagent explores repo, runs opencode research
|
||||
2. Subagent writes findings to `/tmp/findings-{issue}.md`
|
||||
2. Subagent writes findings to `${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{issue}.md`
|
||||
- Note: Use KUGETSU_TEMP_DIR instead of /tmp since opencode blocks /tmp access in headless mode
|
||||
3. Subagent posts findings as issue comment via curl
|
||||
4. User replies with feedback/questions on Gitea
|
||||
5. Subagent (or Hermes) reads reply, continues research
|
||||
@@ -65,7 +66,7 @@ curl -X POST "https://git.example.com/api/v1/repos/{owner}/{repo}/pulls" \
|
||||
|
||||
```json
|
||||
{
|
||||
"goal": "Work on Issue #{N}: {title}\n\nSteps:\n1. Explore ~/repositories/kugetsu\n2. Run opencode research on {specific question}\n3. Write findings to /tmp/findings-{N}.md\n4. cat /tmp/findings-{N}.md to display\n5. Post as issue comment via:\n curl -X POST 'https://git.example.com/api/v1/repos/shoko/kugetsu/issues/{N}/comments' \\\n -H 'Authorization: token ${GITEA_TOKEN}' \\\n -H 'Content-Type: application/json' \\\n -d @/tmp/findings-{N}.md\n6. Ask 2-3 clarifying questions at end for user\n\nToken: abcdefg012345\nRepo: ~/repositories/kugetsu",
|
||||
"goal": "Work on Issue #{N}: {title}\n\nSteps:\n1. Explore ~/repositories/kugetsu\n2. Run opencode research on {specific question}\n3. Write findings to ${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{N}.md\n4. cat the findings file to display\n5. Post as issue comment via:\n curl -X POST 'https://git.example.com/api/v1/repos/shoko/kugetsu/issues/{N}/comments' \\\n -H 'Authorization: token ${GITEA_TOKEN}' \\\n -H 'Content-Type: application/json' \\\n -d @${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{N}.md\n6. Ask 2-3 clarifying questions at end for user\n\nToken: abcdefg012345\nRepo: ~/repositories/kugetsu\n\nNote: Use KUGETSU_TEMP_DIR instead of /tmp since opencode blocks /tmp access in headless mode",
|
||||
"context": "{additional context}",
|
||||
"toolsets": ["terminal"]
|
||||
}
|
||||
|
||||
@@ -150,8 +150,8 @@ result = delegate_task(
|
||||
Steps:
|
||||
1. Read existing docs in ~/repositories/kugetsu/docs/
|
||||
2. Identify message passing mechanisms used
|
||||
3. Write findings to /tmp/findings-4.md
|
||||
4. cat /tmp/findings-4.md
|
||||
3. Write findings to ${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-4.md
|
||||
4. cat findings file
|
||||
5. Post findings as Gitea issue comment
|
||||
|
||||
Gitea: git.example.com
|
||||
@@ -178,8 +178,8 @@ terminal(
|
||||
### 4.3 Posting Findings to Gitea (from subagent)
|
||||
|
||||
```bash
|
||||
# Write findings to temp file first
|
||||
cat > /tmp/findings-4.md << 'EOF'
|
||||
# Write findings to temp file first (use KUGETSU_TEMP_DIR instead of /tmp)
|
||||
cat > ${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-4.md << 'EOF'
|
||||
# Research Findings for Issue #4
|
||||
|
||||
## Message Passing Mechanisms Identified
|
||||
@@ -194,10 +194,10 @@ EOF
|
||||
|
||||
# Post as issue comment
|
||||
curl -X POST "git.example.com/api/v1/repos/shoko/kugetsu/issues/4/comments" \
|
||||
-H "Authorization: token <YOUR_GITEA_TOKEN>" \
|
||||
-H "Authorization: token <YOUR_GITEA_TOKEN>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "User-Agent: Kugetsu-Subagent/1.0" \
|
||||
-d @/tmp/findings-4.md \
|
||||
-d @${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-4.md \
|
||||
--max-time 30 \
|
||||
--retry 3 \
|
||||
--retry-delay 5
|
||||
|
||||
@@ -180,7 +180,8 @@ For Kugetsu's parallel workflow, prefer `terminal(opencode run ...)` for coding
|
||||
|
||||
```bash
|
||||
# One-shot task (blocks until complete)
|
||||
terminal(command="opencode run 'Fix issue #1: add retry logic'", workdir="/tmp/issue-1")
|
||||
# Note: Use KUGETSU_TEMP_DIR instead of /tmp for opencode workdir
|
||||
terminal(command="opencode run 'Fix issue #1: add retry logic'", workdir="${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/issue-1")
|
||||
|
||||
# Background TUI (interactive, returns session_id)
|
||||
terminal(command="opencode", workdir="~/project", background=true, pty=true)
|
||||
@@ -319,7 +320,7 @@ git push --force-with-lease origin my-branch
|
||||
opencode run "Research/fix issue #{N}"
|
||||
|
||||
4. Agent Posts to Gitea
|
||||
curl -X POST .../issues/{N}/comments -d @/tmp/findings-{N}.md
|
||||
curl -X POST .../issues/{N}/comments -d @${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/findings-{N}.md
|
||||
|
||||
5. User Reviews on Gitea
|
||||
Comments on issues/PRs
|
||||
|
||||
@@ -261,7 +261,7 @@ For long-running tasks, SSH and spawn:
|
||||
|
||||
```bash
|
||||
ssh -p 2222 <username>@<host-ip> \
|
||||
"kugetsu start github.com/shoko/kugetsu#11 'Implement feature' && echo 'Task done' | tee /tmp/task.log"
|
||||
"kugetsu start github.com/shoko/kugetsu#11 'Implement feature' && echo 'Task done' | tee \${KUGETSU_TEMP_DIR:-${HOME}/.local/share/opencode/tool-output}/task.log"
|
||||
```
|
||||
|
||||
### Port Forwarding for Web UI
|
||||
|
||||
@@ -97,9 +97,9 @@ cd $REVIEW && opencode run 'Review this PR vs main...'
|
||||
|
||||
```python
|
||||
delegate_task(tasks=[
|
||||
{"goal": "Fix issue #1", "context": "...", "workdir": "/tmp/issue-1"},
|
||||
{"goal": "Fix issue #2", "context": "...", "workdir": "/tmp/issue-2"},
|
||||
{"goal": "Fix issue #3", "context": "...", "workdir": "/tmp/issue-3"},
|
||||
{"goal": "Fix issue #1", "context": "...", "workdir": "${KUGETSU_TEMP_DIR}/issue-1"},
|
||||
{"goal": "Fix issue #2", "context": "...", "workdir": "${KUGETSU_TEMP_DIR}/issue-2"},
|
||||
{"goal": "Fix issue #3", "context": "...", "workdir": "${KUGETSU_TEMP_DIR}/issue-3"},
|
||||
], max_iterations=50)
|
||||
```
|
||||
|
||||
@@ -107,14 +107,15 @@ delegate_task(tasks=[
|
||||
- Cannot use: `delegate_task`, `clarify`, `memory`, `send_message` within subagents
|
||||
- Tasks must be fully self-contained
|
||||
- Exit with Ctrl+C or `process(action="kill")`, NOT `/exit`
|
||||
- Note: Use KUGETSU_TEMP_DIR instead of /tmp since opencode blocks /tmp in headless mode
|
||||
|
||||
### Terminal Parallel Calls
|
||||
|
||||
No hard cap. Fire multiple `opencode run` in parallel:
|
||||
```bash
|
||||
opencode run 'Fix issue #1' --workdir /tmp/issue-1 &
|
||||
opencode run 'Fix issue #2' --workdir /tmp/issue-2 &
|
||||
opencode run 'Fix issue #3' --workdir /tmp/issue-3 &
|
||||
opencode run 'Fix issue #1' --workdir ${KUGETSU_TEMP_DIR}/issue-1 &
|
||||
opencode run 'Fix issue #2' --workdir ${KUGETSU_TEMP_DIR}/issue-2 &
|
||||
opencode run 'Fix issue #3' --workdir ${KUGETSU_TEMP_DIR}/issue-3 &
|
||||
```
|
||||
|
||||
### Schema Constraint
|
||||
|
||||
@@ -49,8 +49,6 @@ A default config file is created during `kugetsu init` with commented examples:
|
||||
| `MAX_CONCURRENT_AGENTS` | 3 | Maximum number of concurrent dev agents |
|
||||
| `KUGETSU_TEMP_DIR` | `~/.local/share/opencode/tool-output` | Temp directory for subagent tool output (useful in headless environments where /tmp is restricted) |
|
||||
| `KUGETSU_VERBOSITY` | `default` | PM agent verbosity level: `verbose`, `default`, or `quiet` |
|
||||
| `QUEUE_DAEMON_INTERVAL_MINUTES` | 5 | How often daemon polls queue (in minutes) |
|
||||
| `QUEUE_CLEANUP_AGE_DAYS` | 7 | Auto-cleanup completed/error items older than N days |
|
||||
|
||||
### Environment Variables for Agents
|
||||
|
||||
@@ -113,10 +111,6 @@ Each issue session gets its own git worktree to prevent conflicts:
|
||||
├── worktrees/
|
||||
│ ├── github.com-shoko-kugetsu-14/ # Isolated workdir for issue #14
|
||||
│ └── github.com-shoko-kugetsu-15/ # Isolated workdir for issue #15
|
||||
├── queue/
|
||||
│ ├── items/ # Queue item JSON files
|
||||
│ ├── daemon.pid # Daemon process ID
|
||||
│ └── daemon.log # Daemon log output
|
||||
└── index.json # Maps session IDs and issue refs to session files
|
||||
```
|
||||
|
||||
@@ -264,17 +258,16 @@ kugetsu destroy --base -y
|
||||
|
||||
### kugetsu delegate `<message>`
|
||||
|
||||
Send a message to the PM agent for task coordination via queue:
|
||||
Send a message to the PM agent for task coordination (fire-and-forget):
|
||||
```bash
|
||||
kugetsu delegate "work on issue #14"
|
||||
kugetsu delegate "review PR #92"
|
||||
```
|
||||
|
||||
- **Always enqueues** (fire-and-forget): returns immediately
|
||||
- Queue daemon polls queue and invokes PM when slots available
|
||||
- Tasks are processed FIFO (first-in-first-out)
|
||||
- Use `kugetsu queue list` to see pending tasks
|
||||
- Use `kugetsu queue-daemon logs` to debug queue processing
|
||||
- Non-blocking: returns immediately, runs in background
|
||||
- PM agent processes the message asynchronously
|
||||
- Uses `KUGETSU_VERBOSITY` env var to control PM agent output verbosity
|
||||
- Log output stored in `~/.kugetsu/logs/delegate-<timestamp>.log`
|
||||
|
||||
### kugetsu logs [n]
|
||||
|
||||
@@ -335,79 +328,35 @@ kugetsu server default github # Set default server
|
||||
kugetsu server get github # Get server URL
|
||||
```
|
||||
|
||||
### kugetsu queue <list|stats|clear>
|
||||
### kugetsu queue <list|enqueue|dequeue|clear>
|
||||
|
||||
Manage task queue for autonomous PM operation:
|
||||
```bash
|
||||
kugetsu queue list # Show queued tasks with status
|
||||
kugetsu queue stats # Show queue statistics (total, pending, notified, completed, error)
|
||||
kugetsu queue clear # Clean up old completed/error items
|
||||
kugetsu queue enqueue <issue-ref> <message> # Manually enqueue a task
|
||||
kugetsu queue list # Show queued tasks
|
||||
kugetsu queue enqueue "task" # Add task to queue
|
||||
kugetsu queue dequeue # Remove next task from queue
|
||||
kugetsu queue clear # Clear all queued tasks
|
||||
```
|
||||
|
||||
**Queue Item States:**
|
||||
- `pending` - Waiting in queue, daemon can pick up
|
||||
- `notified` - PM agent has picked up the task
|
||||
- `completed` - Dev agent finished, PR created
|
||||
- `error` - Timeout or failure
|
||||
|
||||
### kugetsu queue-daemon <start|stop|restart|status|logs>
|
||||
|
||||
Manage the queue daemon background process:
|
||||
```bash
|
||||
kugetsu queue-daemon start # Start daemon in background
|
||||
kugetsu queue-daemon stop # Stop daemon
|
||||
kugetsu queue-daemon restart # Restart daemon
|
||||
kugetsu queue-daemon status # Check if daemon is running
|
||||
kugetsu queue-daemon logs # Show recent daemon logs
|
||||
```
|
||||
|
||||
**Daemon Behavior:**
|
||||
1. Runs at configurable interval (default: 5 minutes)
|
||||
2. Checks if active agents < MAX_CONCURRENT_AGENTS
|
||||
3. Picks 1-N pending items (configurable batch size)
|
||||
4. Forks PM session for each picked item
|
||||
5. PM decides whether to use `start` or `continue`
|
||||
|
||||
**Queue Directory:**
|
||||
```
|
||||
~/.kugetsu/queue/
|
||||
├── items/ # Queue item JSON files
|
||||
│ ├── q_1234567890.json # One file per queued task
|
||||
│ └── q_1234567891.json
|
||||
├── daemon.pid # Daemon process ID
|
||||
├── daemon.lock # Daemon lock file
|
||||
└── daemon.log # Daemon log output
|
||||
```
|
||||
- Queue stored in `~/.kugetsu/queue.json`
|
||||
|
||||
## Workflow Example
|
||||
|
||||
### First-time Setup
|
||||
```bash
|
||||
# Initialize kugetsu (requires TTY)
|
||||
# First-time setup (requires TTY)
|
||||
kugetsu init
|
||||
# Creates: base session + pm-agent session
|
||||
|
||||
# Start the queue daemon (for autonomous operation)
|
||||
kugetsu queue-daemon start
|
||||
```
|
||||
# Start work on issue
|
||||
kugetsu start github.com/shoko/kugetsu#14 "implement feature X"
|
||||
# Creates: worktree at ~/.kugetsu/worktrees/github.com-shoko-kugetsu-14/
|
||||
|
||||
### Normal Workflow
|
||||
```bash
|
||||
# Enqueue tasks via delegate - agents will process them automatically
|
||||
kugetsu delegate "work on issue #14"
|
||||
kugetsu delegate "review PR #92"
|
||||
|
||||
# Check queue status
|
||||
kugetsu queue list # See pending tasks
|
||||
kugetsu queue stats # See statistics
|
||||
|
||||
# Debug queue daemon
|
||||
kugetsu queue-daemon status # Is daemon running?
|
||||
kugetsu queue-daemon logs # See daemon logs
|
||||
|
||||
# Continue work on existing issue
|
||||
# Continue later
|
||||
kugetsu continue github.com/shoko/kugetsu#14 "add tests"
|
||||
|
||||
# Continue again
|
||||
kugetsu continue github.com/shoko/kugetsu#14 "fix failing test"
|
||||
|
||||
# List all sessions
|
||||
kugetsu list
|
||||
|
||||
@@ -418,21 +367,6 @@ kugetsu prune --force
|
||||
kugetsu destroy github.com/shoko/kugetsu#14
|
||||
```
|
||||
|
||||
### Queue Daemon Management
|
||||
```bash
|
||||
# Check if daemon is running
|
||||
kugetsu queue-daemon status
|
||||
|
||||
# View daemon logs for debugging
|
||||
kugetsu queue-daemon logs
|
||||
|
||||
# Restart daemon if needed
|
||||
kugetsu queue-daemon restart
|
||||
|
||||
# Stop daemon
|
||||
kugetsu queue-daemon stop
|
||||
```
|
||||
|
||||
## Headless Operation
|
||||
|
||||
This design solves the headless CLI limitation discovered in Issue #14:
|
||||
|
||||
@@ -84,7 +84,7 @@ When a request comes in:
|
||||
**You:** `kugetsu start <domain>/<user>/<repo>#126 Update README with installation instructions`
|
||||
|
||||
**User:** "Create a file at /tmp/test.txt"
|
||||
**You:** `kugetsu start <domain>/<user>/<repo>#127 Create a file at /tmp/test.txt`
|
||||
**You:** `kugetsu start <domain>/<user>/<repo>#127 Create a file at ${KUGETSU_TEMP_DIR:-~/.local/share/opencode/tool-output}/test.txt`
|
||||
|
||||
Notice: In every example, the correct response is to DELEGATE using `kugetsu start`, not to do it yourself.
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user