From 6eefe5aae320cef1699f68c0ccde8acc25c2727d Mon Sep 17 00:00:00 2001 From: a2a-cloud Date: Tue, 9 Jun 2026 11:50:07 +0000 Subject: [PATCH] a2a-source-edit: remove README.md --- README.md | 145 ------------------------------------------------------ 1 file changed, 145 deletions(-) delete mode 100644 README.md diff --git a/README.md b/README.md deleted file mode 100644 index 2fb8845..0000000 --- a/README.md +++ /dev/null @@ -1,145 +0,0 @@ -# seleniumbase-website-scraper - -Private A2A agent for compliant, authorized website scraping with SeleniumBase/browser automation where available. - -## Safety policy - -Use this agent only for authorized scraping of public or user-owned sites. The agent must not bypass, solve, defeat, or evade CAPTCHAs, anti-bot protections, login gates, paywalls, `robots.txt`, ToS restrictions, or access controls. If a CAPTCHA, anti-bot challenge, login wall, paywall, or access-control indicator is detected, the run stops and returns a manual-intervention/blocked status. - -The agent does not use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention. - -## Skills - -### scrape_website - -Inputs: -- url — absolute http or https URL. -- selectors — mapping of output field names to CSS selectors. -- max_pages — maximum same-origin pages to visit. -- wait_seconds — wait after page open before capture/extraction. -- output_format — json or csv. -- headless — toggle browser headless mode. -- user_agent — optional custom single-line user agent. -- respect_robots — when true, checks robots.txt before fetch and pagination. -- rate_limit_seconds — polite delay between pagination requests. -- screenshot — save PNG screenshots for visited pages. - -Outputs: -- status — ok, invalid_input, blocked_by_robots_txt, manual_intervention_required, browser_runtime_unavailable, or error. -- items — extracted records. -- saved_paths — output files under outputs/seleniumbase-website-scraper/. -- warnings — safety, robots, selector, browser, or runtime warnings. -- screenshots — screenshot file paths when requested. -- pages_visited and html_captures. - -### iterate_scrape_goal (new) - -Parameters: -- start_url, allowed_domains (default same-origin), seed_selectors, max_steps, max_pages, - max_runtime_seconds, wait_seconds, rate_limit_seconds, headless, user_agent, - respect_robots (default true), output_format (json/csv), screenshot. - -Returns fields include: {status, items, pages_visited, html_captures, screenshots, saved_paths, warnings, steps, goal_summary}. - -An LLM-powered planner that iterates scrapes until a goal is satisfied or a safety/stop condition occurs. -Inputs: -- goal — natural-language target (e.g., "Collect quotes text and authors from the first 2 pages"). -- start_url — start page (absolute http/https). Same-origin only for all iterations unless allowed_domains is provided. -- seed_selectors — optional initial CSS selector map (defaults to {"title": "title"}). -- max_steps — planner steps (hard-capped at 3 improvements). -- max_pages — pages per scrape pass (default 1, capped by agent config). -- max_runtime_seconds — total runtime budget across steps. -- wait_seconds, output_format, headless, user_agent, respect_robots, rate_limit_seconds, screenshot — mirror scrape_website. -- allowed_domains — optional whitelist; else restricted to same-origin. - -Outputs: -- status — ok | no_items | blocked_by_robots_txt | manual_intervention_required | browser_runtime_unavailable | error -- items — accumulated unique items across passes -- pages_visited — union of pages visited across passes -- html_captures — captures from passes -- screenshots — when requested -- saved_paths — includes an iter-plan manifest under outputs/seleniumbase-website-scraper/ -- warnings — including planner parsing issues, safety stops, or budget stops -- iterations — number of planner decisions taken -- decisions — array of {iteration, action, reason, confidence, url?, selectors?} - -Behavior and guarantees: -- Uses platform-provided ctx.llm with strict system prompt that forbids any evasion of CAPTCHAs, anti-bot, logins, paywalls, or robots.txt. -- Planner can only propose same-origin URLs; cross-origin plans are rejected and end the loop. -- Each pass runs the deterministic scrape_website skill; stops on any challenge or robots block. -- Produces a compact decisions trail and saves a small manifest under outputs/seleniumbase-website-scraper/. - -## Selector syntax - -Default extraction returns visible text: - -```json -{"title": "h1", "summary": ".article-summary"} -``` - -Supported suffixes: -- ::text — visible text. -- ::html — inner HTML. -- ::attr(name) — attribute extraction, e.g. a.card::attr(href). - -Special pagination selector field names are not emitted as data fields and are used to find the next page: -- next -- _next -- __next__ -- pagination_next -- next_page - -Pagination is restricted to the same scheme and host as the start URL. - -## Example payloads - -Simple one-pass scrape: -```json -{ - "url": "https://example.com/articles", - "selectors": { - "headline": "h2.article-title", - "links": "a.article-link::attr(href)", - "next": "a[rel='next']::attr(href)" - }, - "max_pages": 3, - "wait_seconds": 2, - "output_format": "json", - "headless": true, - "respect_robots": true -} -``` - -Iterative planner: -```json -{ - "goal": "collect at least 30 product tiles with name, price, link", - "url": "https://example.com/store", - "selectors": {"name": ".tile .name", "price": ".tile .price", "href": ".tile a::attr(href)"}, - "min_items": 30, - "max_iterations": 3, - "per_pass_max_pages": 1, - "respect_robots": true -} -``` - -## Smoke test: quotes.toscrape.com - -Goal: "Collect quotes text and authors from the first 2 pages" -- Expected: status=ok, >= 1 item, <= 2 iterations (often 1–2), a saved JSON manifest, no screenshots when disabled. -- Example call (schematic): -```json -{ - "goal": "Collect quotes text and authors from the first 2 pages", - "url": "https://quotes.toscrape.com/", - "selectors": {"text": ".quote .text", "author": ".quote .author", "pagination_next": "li.next a::attr(href)"}, - "min_items": 2, - "max_iterations": 2, - "per_pass_max_pages": 2, - "screenshot": false -} -``` - -## Deployment notes - -The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under /workspace/outputs/seleniumbase-website-scraper/ are durable workspace outputs.