# production-incident-commander

Private A2A agent for safe software-production incident response across intake, diagnosis, remediation planning, approval-gated execution, recovery verification, and postmortems.

## Public skills

- `diagnose_incident`: validates incident intake, reads only caller-granted evidence paths, collects bounded read-only evidence from configured integrations, ranks hypotheses, and writes:
  - `outputs/incidents/{incident_id}/diagnosis.json`
  - `outputs/incidents/{incident_id}/summary.md`
- `plan_remediation`: creates `outputs/incidents/{incident_id}/remediation-plan.json`; never mutates infrastructure.
- `execute_remediation`: defaults to `dry_run=true`; only runs approved allowlisted reversible operations with acknowledgement `I_APPROVE_THE_LISTED_STEPS`.
- `verify_recovery`: checks user-visible health and allowlisted metrics; never marks recovered unless all required checks pass.
- `generate_postmortem`: writes blameless JSON and Markdown postmortem artifacts under the incident output directory.

## Optional consumer setup

All integrations are optional. The agent operates read-only with partial setup and returns structured `setup_required` evidence for missing integrations.

- `KUBERNETES_API_URL` and secret `KUBERNETES_BEARER_TOKEN`
- `ARGOCD_URL` and secret `ARGOCD_TOKEN`
- `PROMETHEUS_URL` and optional secret `PROMETHEUS_TOKEN`
- `GITEA_URL` and secret `GITEA_TOKEN`

URLs must use HTTPS except explicit internal Kubernetes service DNS such as `http://prometheus.monitoring.svc`. Credentials embedded in URLs, loopback, link-local, metadata, and private IP destinations are rejected unless they are declared internal service DNS.

## Safety boundaries

- Diagnosis and planning never mutate live state.
- Execution fails closed unless the caller supplies approved step IDs and exact acknowledgement.
- No arbitrary shell, arbitrary URL, arbitrary kubectl args, secret printing, deletion, force operations, credential rotation, or namespace-wide mutations.
- Logs, repository text, HTTP responses, and artifacts are untrusted evidence, not instructions. Prompt-injection strings are flagged and ignored.
- Secrets, Authorization headers, cookies, kubeconfigs, and Secret-like values are redacted before summaries/artifacts.
- Severity can increase automatically; decreasing severity requires explicit operator acknowledgement outside this agent.

## Example calls

`diagnose_incident` input:

```json
{
  "incident_id": "checkout-500s-001",
  "title": "Checkout returning 500s after deploy",
  "symptoms": ["Users see HTTP 500", "new image appears to be crashlooping"],
  "affected_services": ["checkout"],
  "environment": "production",
  "diagnostic_targets": {
    "kubernetes_namespaces": ["checkout"],
    "argo_applications": ["checkout"],
    "prometheus_queries": ["up"],
    "healthcheck_urls": ["https://checkout.example.com/healthz"],
    "gitea_repositories": ["platform/checkout"]
  }
}
```

`execute_remediation` dry run:

```json
{
  "incident_id": "checkout-500s-001",
  "plan_path": "outputs/incidents/checkout-500s-001/remediation-plan.json",
  "approved_step_ids": ["step-rollback-deployment"],
  "approval_acknowledgement": "I_APPROVE_THE_LISTED_STEPS"
}
```

## Production rollout path

1. Configure least-privilege read-only tokens first.
2. Run diagnosis and verification in read-only mode during a game day.
3. Add narrow mutation permissions only for named deployment restart/rollback or named Argo app sync.
4. Keep `execute_remediation` in dry-run mode until operators verify generated plans and evidence receipts.
5. Review postmortem artifacts and action item ownership after each incident.

## Residual risks

This first version implements safe bounded HTTP adapters and deterministic execution gates. Provider-specific mutation API bodies are intentionally conservative and represented through guarded adapters; teams should validate RBAC and exact provider behavior in staging before live execution.