write skills/llm-iterative-scrape/SKILL.md

skills/llm-iterative-scrape/SKILL.md

@@ -0,0 +1,31 @@
+---
+name: llm-iterative-scrape
+description: "Iterative compliant scraping workflow: plan -> bounded browser step -> evaluate -> refine until goal satisfied or safety/budget stops."
+---
+# llm-iterative-scrape
+
+Use this skill for iterative, goal-driven scraping on authorized public or user-owned sites.
+
+Loop:
+1) Translate the natural-language goal into a minimal set of CSS selectors and a short plan for ONE bounded browser step.
+2) Execute exactly one small browser step with a polite wait/rate limit via the host agent's primitives.
+3) Evaluate observations vs. the goal. Refine minimally. Stop when satisfied or when any safety/budget condition triggers.
+
+Hard safety constraints:
+- Never solve or route around CAPTCHAs/anti-bot.
+- Never log in or pass paywalls.
+- Restrict pagination to same-origin as the start URL by default; also respect provided allowed_domains.
+- When `respect_robots` is true, treat any robots.txt disallow or failure-to-check as a stop condition.
+
+Selectors guidance:
+- Use compact CSS selectors. Suffix conventions: `::text`, `::html`, `::attr(name)`.
+- Use one pagination field: `pagination_next` or `next` for the next-page `href`.
+- Prefer domain-agnostic patterns like `a[rel='next']::attr(href)` or `li.next a::attr(href)`.
+
+Planning output contract:
+Return strictly compact JSON only: {"plan_summary": str, "selectors": {str:str}, "continue": bool, "notes": str}. Keep under ~600 chars when possible.
+
+Evaluation guidance:
+- If extracted data matches the requested fields and count threshold, set continue=false and summarize.
+- If pagination is needed for more pages and a next link exists, keep continue=true.
+- On any safety trigger (captcha/login/paywall/robots), set continue=false and summarize the stop reason.