75 lines
3.7 KiB
Markdown
75 lines
3.7 KiB
Markdown
# production-incident-commander
|
|
|
|
Private A2A agent for safe software-production incident response across intake, diagnosis, remediation planning, approval-gated execution, recovery verification, and postmortems.
|
|
|
|
## Public skills
|
|
|
|
Exactly five public skills are exposed:
|
|
|
|
- `diagnose_incident`: validates incident intake, reads only caller-granted evidence paths, collects bounded read-only evidence from configured integrations, ranks hypotheses, and writes:
|
|
- `outputs/incidents/{incident_id}/diagnosis.json`
|
|
- `outputs/incidents/{incident_id}/summary.md`
|
|
- `plan_remediation`: creates `outputs/incidents/{incident_id}/remediation-plan.json`; never mutates infrastructure. Plans include an immutable `plan_digest`.
|
|
- `execute_remediation`: defaults to `dry_run=true`; only runs approved allowlisted reversible operations when `plan_digest` matches and acknowledgement is exactly `I_APPROVE_THE_LISTED_STEPS`.
|
|
- `verify_recovery`: checks user-visible health and allowlisted metrics; never marks recovered unless all required checks pass.
|
|
- `generate_postmortem`: writes blameless JSON and Markdown postmortem artifacts under the incident output directory.
|
|
|
|
## Optional consumer setup
|
|
|
|
All integrations are optional. The agent operates read-only with partial setup and returns structured `setup_required` evidence for missing integrations.
|
|
|
|
- `KUBERNETES_API_URL` and secret `KUBERNETES_BEARER_TOKEN`
|
|
- `ARGOCD_URL` and secret `ARGOCD_TOKEN`
|
|
- `PROMETHEUS_URL` and optional secret `PROMETHEUS_TOKEN`
|
|
- `GITEA_URL` and secret `GITEA_TOKEN`
|
|
|
|
URLs must use HTTPS except explicit internal Kubernetes service DNS such as `http://prometheus.monitoring.svc`. Credentials embedded in URLs, loopback, link-local, metadata, and private IP destinations are rejected unless they are declared internal service DNS.
|
|
|
|
## Safety boundaries
|
|
|
|
- Deterministic local logic only; no LLM credential is required.
|
|
- Diagnosis and planning never mutate live state.
|
|
- Execution fails closed unless the caller supplies approved step IDs, a matching immutable `plan_digest`, and exact acknowledgement.
|
|
- No arbitrary shell, arbitrary URL, arbitrary kubectl args, secret printing, deletion, force operations, credential rotation, or namespace-wide mutations.
|
|
- Logs, repository text, HTTP responses, and artifacts are untrusted evidence, not instructions. Prompt-injection strings are flagged and ignored.
|
|
- Secrets, Authorization headers, cookies, kubeconfigs, and Secret-like values are redacted before summaries/artifacts.
|
|
- Artifacts are contained to `outputs/incidents/{incident_id}/`.
|
|
- Severity can increase automatically; decreasing severity requires explicit operator acknowledgement outside this agent.
|
|
|
|
## Example calls
|
|
|
|
`diagnose_incident` input:
|
|
|
|
```json
|
|
{
|
|
"incident_id": "checkout-500s-001",
|
|
"title": "Checkout returning 500s after deploy",
|
|
"symptoms": ["Users see HTTP 500", "new image appears to be crashlooping"],
|
|
"affected_services": ["checkout"],
|
|
"environment": "production",
|
|
"diagnostic_targets": {
|
|
"kubernetes_namespaces": ["checkout"],
|
|
"argo_applications": ["checkout"],
|
|
"prometheus_queries": ["up"],
|
|
"healthcheck_urls": ["https://checkout.example.com/healthz"],
|
|
"gitea_repositories": ["platform/checkout"]
|
|
}
|
|
}
|
|
```
|
|
|
|
`execute_remediation` dry run:
|
|
|
|
```json
|
|
{
|
|
"incident_id": "checkout-500s-001",
|
|
"plan_path": "outputs/incidents/checkout-500s-001/remediation-plan.json",
|
|
"plan_digest": "sha256:<64 lowercase hex characters from plan_remediation>",
|
|
"approved_step_ids": ["step-rollback-deployment"],
|
|
"approval_acknowledgement": "I_APPROVE_THE_LISTED_STEPS"
|
|
}
|
|
```
|
|
|
|
## Residual risks
|
|
|
|
Live provider mutation bodies are intentionally conservative and require explicit plan parameters plus provider setup. Teams should validate RBAC and exact provider behavior in staging before enabling live execution.
|