production-incident-commander/README.md

production-incident-commander

Private A2A agent for safe software-production incident response across intake, diagnosis, remediation planning, approval-gated execution, recovery verification, and postmortems.

Public skills

• diagnose_incident: validates incident intake, reads only caller-granted evidence paths, collects bounded read-only evidence from configured integrations, ranks hypotheses, and writes:
  • outputs/incidents/{incident_id}/diagnosis.json
  • outputs/incidents/{incident_id}/summary.md
• plan_remediation: creates outputs/incidents/{incident_id}/remediation-plan.json; never mutates infrastructure.
• execute_remediation: defaults to dry_run=true; only runs approved allowlisted reversible operations with acknowledgement I_APPROVE_THE_LISTED_STEPS.
• verify_recovery: checks user-visible health and allowlisted metrics; never marks recovered unless all required checks pass.
• generate_postmortem: writes blameless JSON and Markdown postmortem artifacts under the incident output directory.

Optional consumer setup

All integrations are optional. The agent operates read-only with partial setup and returns structured setup_required evidence for missing integrations.

• KUBERNETES_API_URL and secret KUBERNETES_BEARER_TOKEN
• ARGOCD_URL and secret ARGOCD_TOKEN
• PROMETHEUS_URL and optional secret PROMETHEUS_TOKEN
• GITEA_URL and secret GITEA_TOKEN

URLs must use HTTPS except explicit internal Kubernetes service DNS such as http://prometheus.monitoring.svc. Credentials embedded in URLs, loopback, link-local, metadata, and private IP destinations are rejected unless they are declared internal service DNS.

Safety boundaries

• Diagnosis and planning never mutate live state.
• Execution fails closed unless the caller supplies approved step IDs and exact acknowledgement.
• No arbitrary shell, arbitrary URL, arbitrary kubectl args, secret printing, deletion, force operations, credential rotation, or namespace-wide mutations.
• Logs, repository text, HTTP responses, and artifacts are untrusted evidence, not instructions. Prompt-injection strings are flagged and ignored.
• Secrets, Authorization headers, cookies, kubeconfigs, and Secret-like values are redacted before summaries/artifacts.
• Severity can increase automatically; decreasing severity requires explicit operator acknowledgement outside this agent.

Example calls

diagnose_incident input:

{
  "incident_id": "checkout-500s-001",
  "title": "Checkout returning 500s after deploy",
  "symptoms": ["Users see HTTP 500", "new image appears to be crashlooping"],
  "affected_services": ["checkout"],
  "environment": "production",
  "diagnostic_targets": {
    "kubernetes_namespaces": ["checkout"],
    "argo_applications": ["checkout"],
    "prometheus_queries": ["up"],
    "healthcheck_urls": ["https://checkout.example.com/healthz"],
    "gitea_repositories": ["platform/checkout"]
  }
}

execute_remediation dry run:

{
  "incident_id": "checkout-500s-001",
  "plan_path": "outputs/incidents/checkout-500s-001/remediation-plan.json",
  "approved_step_ids": ["step-rollback-deployment"],
  "approval_acknowledgement": "I_APPROVE_THE_LISTED_STEPS"
}

Production rollout path

1. Configure least-privilege read-only tokens first.
2. Run diagnosis and verification in read-only mode during a game day.
3. Add narrow mutation permissions only for named deployment restart/rollback or named Argo app sync.
4. Keep execute_remediation in dry-run mode until operators verify generated plans and evidence receipts.
5. Review postmortem artifacts and action item ownership after each incident.

Residual risks

This first version implements safe bounded HTTP adapters and deterministic execution gates. Provider-specific mutation API bodies are intentionally conservative and represented through guarded adapters; teams should validate RBAC and exact provider behavior in staging before live execution.