From f8348f6e58490292625178145af635509e7a31cd Mon Sep 17 00:00:00 2001 From: a2a-cloud Date: Mon, 8 Jun 2026 22:41:40 +0000 Subject: [PATCH] edit README.md --- README.md | 98 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 61 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 0b08366..5102770 100644 --- a/README.md +++ b/README.md @@ -6,33 +6,48 @@ Private A2A agent for compliant, authorized website scraping with SeleniumBase/b Use this agent only for authorized scraping of public or user-owned sites. The agent must not bypass, solve, defeat, or evade CAPTCHAs, anti-bot protections, login gates, paywalls, `robots.txt`, ToS restrictions, or access controls. If a CAPTCHA, anti-bot challenge, login wall, paywall, or access-control indicator is detected, the run stops and returns a manual-intervention/blocked status. -The agent does **not** use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention. +The agent does not use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention. -## Skill +## Skills -### `scrape_website` +### scrape_website Inputs: - -- `url` — absolute `http` or `https` URL. -- `selectors` — mapping of output field names to CSS selectors. -- `max_pages` — maximum same-origin pages to visit. -- `wait_seconds` — wait after page open before capture/extraction. -- `output_format` — `json` or `csv`. -- `headless` — toggle browser headless mode. -- `user_agent` — optional custom single-line user agent. -- `respect_robots` — when true, checks `robots.txt` before fetch and pagination. -- `rate_limit_seconds` — polite delay between pagination requests. -- `screenshot` — save PNG screenshots for visited pages. +- url — absolute http or https URL. +- selectors — mapping of output field names to CSS selectors. +- max_pages — maximum same-origin pages to visit. +- wait_seconds — wait after page open before capture/extraction. +- output_format — json or csv. +- headless — toggle browser headless mode. +- user_agent — optional custom single-line user agent. +- respect_robots — when true, checks robots.txt before fetch and pagination. +- rate_limit_seconds — polite delay between pagination requests. +- screenshot — save PNG screenshots for visited pages. Outputs: +- status — ok, invalid_input, blocked_by_robots_txt, manual_intervention_required, browser_runtime_unavailable, or error. +- items — extracted records. +- saved_paths — output files under outputs/seleniumbase-website-scraper/. +- warnings — safety, robots, selector, browser, or runtime warnings. +- screenshots — screenshot file paths when requested. +- pages_visited and html_captures. -- `status` — `ok`, `invalid_input`, `blocked_by_robots_txt`, `manual_intervention_required`, `browser_runtime_unavailable`, or `error`. -- `items` — extracted records. -- `saved_paths` — output files under `outputs/seleniumbase-website-scraper/`. -- `warnings` — safety, robots, selector, browser, or runtime warnings. -- `screenshots` — screenshot file paths when requested. -- `pages_visited` and `html_captures`. +### iterate_scrape_goal (new) + +An LLM-powered planner that iterates scrapes until a goal is satisfied or a safety/stop condition occurs. +- goal — natural-language target (e.g., "collect at least 50 product cards with title, price, href"). +- url — start page (absolute http/https). Same-origin only for all iterations. +- selectors — optional initial CSS selector map (defaults to {"title": "title"}). +- min_items — stop when this many items have been collected. +- max_iterations — planner budget (default 5). +- per_pass_max_pages — pages per scrape pass (default 1, capped by agent config). +- Other browser and safety flags mirror scrape_website. + +Behavior and guarantees: +- Uses platform-provided ctx.llm with strict system prompt that forbids any evasion of CAPTCHAs, anti-bot, logins, paywalls, or robots.txt. +- Planner can only propose same-origin URLs and CSS selectors. Cross-origin plans are rejected. +- Each pass runs the deterministic scrape_website skill; stops on any challenge or robots block. +- Produces a compact decisions trail and saves a small manifest under outputs/seleniumbase-website-scraper/. ## Selector syntax @@ -43,23 +58,22 @@ Default extraction returns visible text: ``` Supported suffixes: - -- `::text` — visible text. -- `::html` — inner HTML. -- `::attr(name)` — attribute extraction, e.g. `a.card::attr(href)`. +- ::text — visible text. +- ::html — inner HTML. +- ::attr(name) — attribute extraction, e.g. a.card::attr(href). Special pagination selector field names are not emitted as data fields and are used to find the next page: - -- `next` -- `_next` -- `__next__` -- `pagination_next` -- `next_page` +- next +- _next +- __next__ +- pagination_next +- next_page Pagination is restricted to the same scheme and host as the start URL. -## Example payload +## Example payloads +Simple one-pass scrape: ```json { "url": "https://example.com/articles", @@ -72,16 +86,26 @@ Pagination is restricted to the same scheme and host as the start URL. "wait_seconds": 2, "output_format": "json", "headless": true, - "user_agent": "", - "respect_robots": true, - "rate_limit_seconds": 2, - "screenshot": false + "respect_robots": true +} +``` + +Iterative planner: +```json +{ + "goal": "collect at least 30 product tiles with name, price, link", + "url": "https://example.com/store", + "selectors": {"name": ".tile .name", "price": ".tile .price", "href": ".tile a::attr(href)"}, + "min_items": 30, + "max_iterations": 4, + "per_pass_max_pages": 1, + "respect_robots": true } ``` ## Smoke test payload -Use this payload to verify the scraper against the public demo site `https://quotes.toscrape.com/` while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when `screenshot` is `true`. +Use this payload to verify the scraper against the public demo site https://quotes.toscrape.com/ while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when screenshot is true. ```json { @@ -105,4 +129,4 @@ Use this payload to verify the scraper against the public demo site `https://quo ## Deployment notes -The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under `/workspace/outputs/seleniumbase-website-scraper/` are durable workspace outputs. +The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under /workspace/outputs/seleniumbase-website-scraper/ are durable workspace outputs.