seleniumbase-website-scraper/skills/llm-iterative-scrape/SKILL.md

name, description

name	description
llm-iterative-scrape	Iterative compliant scraping workflow: plan -> bounded browser step -> evaluate -> refine until goal satisfied or safety/budget stops.

llm-iterative-scrape

Use this skill for iterative, goal-driven scraping on authorized public or user-owned sites.

Loop:

1. Translate the natural-language goal into a minimal set of CSS selectors and a short plan for ONE bounded browser step.
2. Execute exactly one small browser step with a polite wait/rate limit via the host agent's primitives.
3. Evaluate observations vs. the goal. Refine minimally. Stop when satisfied or when any safety/budget condition triggers.

Hard safety constraints:

• Never solve or route around CAPTCHAs/anti-bot.
• Never log in or pass paywalls.
• Restrict pagination to same-origin as the start URL by default; also respect provided allowed_domains.
• When respect_robots is true, treat any robots.txt disallow or failure-to-check as a stop condition.

Selectors guidance:

• Use compact CSS selectors. Suffix conventions: ::text, ::html, ::attr(name).
• Use one pagination field: pagination_next or next for the next-page href.
• Prefer domain-agnostic patterns like a[rel='next']::attr(href) or li.next a::attr(href).

Planning output contract: Return strictly compact JSON only: {"plan_summary": str, "selectors": {str:str}, "continue": bool, "notes": str}. Keep under ~600 chars when possible.

Evaluation guidance:

• If extracted data matches the requested fields and count threshold, set continue=false and summarize.
• If pagination is needed for more pages and a next link exists, keep continue=true.
• On any safety trigger (captcha/login/paywall/robots), set continue=false and summarize the stop reason.