write README.md

README.md

@@ -0,0 +1,108 @@
+# seleniumbase-website-scraper
+
+Private A2A agent for compliant, authorized website scraping with SeleniumBase/browser automation where available.
+
+## Safety policy
+
+Use this agent only for authorized scraping of public or user-owned sites. The agent must not bypass, solve, defeat, or evade CAPTCHAs, anti-bot protections, login gates, paywalls, `robots.txt`, ToS restrictions, or access controls. If a CAPTCHA, anti-bot challenge, login wall, paywall, or access-control indicator is detected, the run stops and returns a manual-intervention/blocked status.
+
+The agent does **not** use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention.
+
+## Skill
+
+### `scrape_website`
+
+Inputs:
+
+- `url` — absolute `http` or `https` URL.
+- `selectors` — mapping of output field names to CSS selectors.
+- `max_pages` — maximum same-origin pages to visit.
+- `wait_seconds` — wait after page open before capture/extraction.
+- `output_format` — `json` or `csv`.
+- `headless` — toggle browser headless mode.
+- `user_agent` — optional custom single-line user agent.
+- `respect_robots` — when true, checks `robots.txt` before fetch and pagination.
+- `rate_limit_seconds` — polite delay between pagination requests.
+- `screenshot` — save PNG screenshots for visited pages.
+
+Outputs:
+
+- `status` — `ok`, `invalid_input`, `blocked_by_robots_txt`, `manual_intervention_required`, `browser_runtime_unavailable`, or `error`.
+- `items` — extracted records.
+- `saved_paths` — output files under `outputs/seleniumbase-website-scraper/`.
+- `warnings` — safety, robots, selector, browser, or runtime warnings.
+- `screenshots` — screenshot file paths when requested.
+- `pages_visited` and `html_captures`.
+
+## Selector syntax
+
+Default extraction returns visible text:
+
+```json
+{"title": "h1", "summary": ".article-summary"}
+```
+
+Supported suffixes:
+
+- `::text` — visible text.
+- `::html` — inner HTML.
+- `::attr(name)` — attribute extraction, e.g. `a.card::attr(href)`.
+
+Special pagination selector field names are not emitted as data fields and are used to find the next page:
+
+- `next`
+- `_next`
+- `__next__`
+- `pagination_next`
+- `next_page`
+
+Pagination is restricted to the same scheme and host as the start URL.
+
+## Example payload
+
+```json
+{
+  "url": "https://example.com/articles",
+  "selectors": {
+    "headline": "h2.article-title",
+    "links": "a.article-link::attr(href)",
+    "next": "a[rel='next']::attr(href)"
+  },
+  "max_pages": 3,
+  "wait_seconds": 2,
+  "output_format": "json",
+  "headless": true,
+  "user_agent": "",
+  "respect_robots": true,
+  "rate_limit_seconds": 2,
+  "screenshot": false
+}
+```
+
+## Smoke test payload
+
+Use this payload to verify the scraper against the public demo site `https://quotes.toscrape.com/` while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when `screenshot` is `true`.
+
+```json
+{
+  "url": "https://quotes.toscrape.com/",
+  "selectors": {
+    "quotes": ".quote",
+    "text": ".text",
+    "author": ".author",
+    "tags": ".tag"
+  },
+  "max_pages": 1,
+  "wait_seconds": 1,
+  "output_format": "json",
+  "headless": true,
+  "user_agent": "",
+  "respect_robots": true,
+  "rate_limit_seconds": 1,
+  "screenshot": false
+}
+```
+
+## Deployment notes
+
+The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under `/workspace/outputs/seleniumbase-website-scraper/` are durable workspace outputs.