121 lines
4.8 KiB
Markdown
121 lines
4.8 KiB
Markdown
---
|
|
name: agent-quality-review
|
|
description: Review a generated a2a-pack agent before sandbox testing or deploy. Use to catch broken cards, missing schemas, fake tools, unwired DeepAgents skills, missing runtime mirrors, unsafe IO, and stale live Agent Card schemas.
|
|
---
|
|
# Agent Quality Review
|
|
|
|
Before `test_agent_in_sandbox` and before deploy, review the generated project
|
|
as source code, not just as text.
|
|
|
|
## Required Files
|
|
|
|
Every generated project needs:
|
|
|
|
- `agent.py`
|
|
- `a2a.yaml`
|
|
- `requirements.txt`
|
|
- Optional `skills/<skill-name>/SKILL.md` bundles for nontrivial DeepAgents
|
|
behavior
|
|
- Optional `frontend/` source when the agent ships a packed app
|
|
|
|
Run `list_agent_files` and read the files that matter. If the project was
|
|
deployed before, use `sync_agent_workspace_from_repo` before editing when the
|
|
builder needs to continue from the current managed repo source.
|
|
|
|
## Agent.py Review
|
|
|
|
Check these items:
|
|
|
|
- Public `@skill` methods are async and have `RunContext[...]` plus typed JSON
|
|
arguments.
|
|
- The public schema is small and stable. No unbounded command strings, hidden
|
|
prompt fragments, or provider credentials as public inputs.
|
|
- The implementation reads `ctx.llm` for both `CALLER_PROVIDED` and
|
|
`PLATFORM` LLM provisioning. `PLATFORM_OR_CALLER_PROVIDED` is acceptable
|
|
only for trusted platform/meta agents and must also read `ctx.llm`.
|
|
- Hosted generated/user agents should default to
|
|
`llm_provisioning = LLMProvisioning.PLATFORM` and
|
|
`Pricing(..., caller_pays_llm=False, ...)`. Use
|
|
`LLMProvisioning.CALLER_PROVIDED` only when the user explicitly wants BYOK or
|
|
caller-paid inference. In both modes, the code must use `ctx.llm` and never
|
|
read LiteLLM/provider keys directly.
|
|
- `Pricing` contains only `price_per_call_usd`, `caller_pays_llm`, and `notes`.
|
|
Reject `runtime.pricing.compute`, `runtime.pricing.total_usd`, `compute=`, or
|
|
`total_usd=` in `agent.py`; those are derived platform fields.
|
|
- Any code path that builds `ChatOpenAI` checks `ctx.llm.api_key` first or
|
|
returns a clear setup/config result before the model constructor runs.
|
|
- Any use of DeepAgents file tools passes `backend=ctx.workspace_backend()`.
|
|
- Any project skills are seeded into the backend and passed with
|
|
`skills=skill_sources or None`.
|
|
- Custom subagents that need project skills include their own `skills` field.
|
|
- Deterministic tools do exact work; no fake canned-response tools.
|
|
- File-producing skills call `ctx.write_artifact` and `ctx.emit_artifact`.
|
|
- Commands that create downloadable outputs run through `ctx.workspace_shell`
|
|
or `ctx.workspace_python`; `/workspace` writes persist directly and other
|
|
changed sandbox rootfs files are mirrored under `outputs/rootfs-captures/...`.
|
|
Plain subprocesses in the agent container are not durable workspace writes.
|
|
- External calls, secrets, workspace access, pricing, and resources are
|
|
declared on the class.
|
|
|
|
## A2A YAML Review
|
|
|
|
Mirror deployment-only details in `a2a.yaml` because the platform reads it
|
|
before importing user code:
|
|
|
|
```yaml
|
|
name: research-agent
|
|
version: 0.1.0
|
|
entrypoint: agent:ResearchAgent
|
|
expose:
|
|
public: true
|
|
runtime:
|
|
apt_packages: [ffmpeg]
|
|
resources:
|
|
cpu: "2"
|
|
memory: 2Gi
|
|
max_runtime_seconds: 900
|
|
```
|
|
|
|
Use `runtime.apt_packages` only for Debian system binaries. Python packages
|
|
belong in `requirements.txt`.
|
|
|
|
If a packed frontend is declared, verify the manifest and files line up:
|
|
|
|
```yaml
|
|
frontend:
|
|
path: frontend
|
|
build: npm run build
|
|
dist: dist
|
|
mount: /app
|
|
auth: inherit
|
|
```
|
|
|
|
For React/Vite frontends, expect `frontend/package.json`,
|
|
`frontend/vite.config.js`, `frontend/src/App.jsx`, and `frontend/src/a2a.js`.
|
|
For static frontends, expect `frontend/dist/index.html`. The browser code must
|
|
not contain platform secrets, provider keys, LiteLLM keys, hard-coded deployment
|
|
URLs, or private stack details. It should load `/app/config.json`, use
|
|
`/app/a2a-client.js` or generated config endpoints, and call only the public
|
|
`@skill` schemas.
|
|
|
|
## Sandbox And Live Card
|
|
|
|
Always run `test_agent_in_sandbox(name)` before deploy. Treat nonzero exit as
|
|
source failure, then read stderr, edit, and rerun.
|
|
|
|
When a frontend is present, inspect the sandbox output's `a2a frontend info`.
|
|
For React apps, make sure `frontend/package.json` defines a build script and
|
|
the deployed build process can produce `frontend/dist/index.html`.
|
|
|
|
After `cp_deploy_tarball`, compare `live_skills[].input_schema` with the
|
|
actual public `@skill` signatures. If a live schema is missing an argument or
|
|
shows an old shape, refresh or redeploy until the live Agent Card matches the
|
|
source.
|
|
|
|
## Minimal Good Result
|
|
|
|
For most generated agents, a good result is one public A2A `@skill`, one
|
|
workspace-backed DeepAgent, one or two internal DeepAgents skills, and a few
|
|
small deterministic tools. Resist broad endpoint sets unless the user asked
|
|
for a multi-operation agent.
|