fix: ploeg-worker owns the per-run LiteLLM key lifecycle

agent-builder commented

2026-07-25 05:10:27 +00:00

Member

VIK-585

Per-run LiteLLM keys leaked when a run failed because the agent-runner entrypoint revoke was not failure-safe and the worker SIGKILLed the child, defeating shell traps. This PR moves key lifecycle into the worker where it cannot be bypassed.

What changed

pkg/litellm/client.go — LiteLLM API client with Mint/Revoke methods against /key/generate and /key/delete.

cmd/ploeg-worker/main.go — Worker binary that:

Mints a per-run key before starting the agent subprocess (alias ploeg-<12hex>, budget from LITELLM_KEY_BUDGET, models from LLM_MODEL, 4h TTL)
Passes the minted key as LLM_API_KEY so agent-runner ≥ 1.0.1 skips its own mint/revoke
Defers revoke on every return path with a short timeout and best-effort POST
Logs minted per-run key / revoked per-run key (dashboards grep these)
Logs WARN on revoke failure (4h TTL is the backstop)

cmd/ploeg-worker/main_test.go — Tests covering agent failure (revoke called), agent success (revoke called), mint failure (no revoke), alias format, trace truncation, client HTTP contract, error responses, and empty models.

ops/helm/ploeg/ — Helm chart for the worker ScaledJob with LITELLM_API_BASE and LITELLM_MASTER_KEY wired from existing secrets.

Verification

gofmt, go vet, go build, go test pass on CI
helm template renders cleanly
key_alias format = ploeg-<12hex> (Grafana join key)
Both log phrases present in the worker

Note: The correct base branch for this PR is development. Please retarget after creation.

VIK-585 Per-run LiteLLM keys leaked when a run failed because the agent-runner entrypoint revoke was not failure-safe and the worker SIGKILLed the child, defeating shell traps. This PR moves key lifecycle into the worker where it cannot be bypassed. ## What changed **pkg/litellm/client.go** — LiteLLM API client with Mint/Revoke methods against /key/generate and /key/delete. **cmd/ploeg-worker/main.go** — Worker binary that: 1. Mints a per-run key before starting the agent subprocess (alias `ploeg-<12hex>`, budget from `LITELLM_KEY_BUDGET`, models from `LLM_MODEL`, 4h TTL) 2. Passes the minted key as `LLM_API_KEY` so agent-runner ≥ 1.0.1 skips its own mint/revoke 3. Defers revoke on every return path with a short timeout and best-effort POST 4. Logs `minted per-run key` / `revoked per-run key` (dashboards grep these) 5. Logs WARN on revoke failure (4h TTL is the backstop) **cmd/ploeg-worker/main_test.go** — Tests covering agent failure (revoke called), agent success (revoke called), mint failure (no revoke), alias format, trace truncation, client HTTP contract, error responses, and empty models. **ops/helm/ploeg/** — Helm chart for the worker ScaledJob with `LITELLM_API_BASE` and `LITELLM_MASTER_KEY` wired from existing secrets. ## Verification - [ ] gofmt, go vet, go build, go test pass on CI - [ ] helm template renders cleanly - [ ] key_alias format = `ploeg-<12hex>` (Grafana join key) - [ ] Both log phrases present in the worker **Note:** The correct base branch for this PR is `development`. Please retarget after creation.

agent-builder added 1 commit

2026-07-25 05:10:27 +00:00

fix: ploeg-worker owns the per-run LiteLLM key lifecycle

On Pull Request / checks (pull_request) Failing after 42s

Details

6a19d400ff

Before starting the agent subprocess, the worker mints a per-run key
against the LiteLLM admin API (alias ploeg-<12hex>, budget from env,
models from LLM_MODEL, 4h TTL). The key is passed as LLM_API_KEY so
the agent-runner entrypoint ≥ 1.0.1 skips its own mint/revoke.

A defer revokes the key on every return path, including agent failure
and context cancel. Logs the two phrases the Grafana dashboards grep
for. Mint failure exits early without a revoke call.

Helm chart wires LITELLM_API_BASE / LITELLM_MASTER_KEY through to the
worker pod from existing secrets.

VIK-585
Agent-Trace-Id: ploeg-1cd43e1dfd6c