bobseares-2/seleniumbase-website-scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

edit README.md

Browse Source
This commit is contained in:
a2a-cloud
2026-06-08 22:41:40 +00:00
parent 1d6b923fa6
commit f8348f6e58
1 changed files with 61 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
README.md
View File

@@ -6,33 +6,48 @@ Private A2A agent for compliant, authorized website scraping with SeleniumBase/b
Use this agent only for authorized scraping of public or user-owned sites. The agent must not bypass, solve, defeat, or evade CAPTCHAs, anti-bot protections, login gates, paywalls, `robots.txt`, ToS restrictions, or access controls. If a CAPTCHA, anti-bot challenge, login wall, paywall, or access-control indicator is detected, the run stops and returns a manual-intervention/blocked status.
The agent does **not** use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention.
The agent does not use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention.
## Skill
## Skills
### `scrape_website`
### scrape_website
Inputs:
- `url` — absolute `http` or `https` URL.
- `selectors` — mapping of output field names to CSS selectors.
- `max_pages` — maximum same-origin pages to visit.
- `wait_seconds` — wait after page open before capture/extraction.
- `output_format` — `json` or `csv`.
- `headless` — toggle browser headless mode.
- `user_agent` — optional custom single-line user agent.
- `respect_robots` — when true, checks `robots.txt` before fetch and pagination.
- `rate_limit_seconds` — polite delay between pagination requests.
- `screenshot` — save PNG screenshots for visited pages.
- url — absolute http or https URL.
- selectors — mapping of output field names to CSS selectors.
- max_pages — maximum same-origin pages to visit.
- wait_seconds — wait after page open before capture/extraction.
- output_format — json or csv.
- headless — toggle browser headless mode.
- user_agent — optional custom single-line user agent.
- respect_robots — when true, checks robots.txt before fetch and pagination.
- rate_limit_seconds — polite delay between pagination requests.
- screenshot — save PNG screenshots for visited pages.
Outputs:
- status — ok, invalid_input, blocked_by_robots_txt, manual_intervention_required, browser_runtime_unavailable, or error.
- items — extracted records.
- saved_paths — output files under outputs/seleniumbase-website-scraper/.
- warnings — safety, robots, selector, browser, or runtime warnings.
- screenshots — screenshot file paths when requested.
- pages_visited and html_captures.
- `status` — `ok`, `invalid_input`, `blocked_by_robots_txt`, `manual_intervention_required`, `browser_runtime_unavailable`, or `error`.
- `items` — extracted records.
- `saved_paths` — output files under `outputs/seleniumbase-website-scraper/`.
- `warnings` — safety, robots, selector, browser, or runtime warnings.
- `screenshots` — screenshot file paths when requested.
- `pages_visited` and `html_captures`.
### iterate_scrape_goal (new)
An LLM-powered planner that iterates scrapes until a goal is satisfied or a safety/stop condition occurs.
- goal — natural-language target (e.g., "collect at least 50 product cards with title, price, href").
- url — start page (absolute http/https). Same-origin only for all iterations.
- selectors — optional initial CSS selector map (defaults to {"title": "title"}).
- min_items — stop when this many items have been collected.
- max_iterations — planner budget (default 5).
- per_pass_max_pages — pages per scrape pass (default 1, capped by agent config).
- Other browser and safety flags mirror scrape_website.
Behavior and guarantees:
- Uses platform-provided ctx.llm with strict system prompt that forbids any evasion of CAPTCHAs, anti-bot, logins, paywalls, or robots.txt.
- Planner can only propose same-origin URLs and CSS selectors. Cross-origin plans are rejected.
- Each pass runs the deterministic scrape_website skill; stops on any challenge or robots block.
- Produces a compact decisions trail and saves a small manifest under outputs/seleniumbase-website-scraper/.
## Selector syntax
@@ -43,23 +58,22 @@ Default extraction returns visible text:
```
Supported suffixes:
- `::text` — visible text.
- `::html` — inner HTML.
- `::attr(name)` — attribute extraction, e.g. `a.card::attr(href)`.
- ::text — visible text.
- ::html — inner HTML.
- ::attr(name) — attribute extraction, e.g. a.card::attr(href).
Special pagination selector field names are not emitted as data fields and are used to find the next page:
- `next`
- `_next`
- `__next__`
- `pagination_next`
- `next_page`
- next
- _next
- __next__
- pagination_next
- next_page
Pagination is restricted to the same scheme and host as the start URL.
## Example payload
## Example payloads
Simple one-pass scrape:
```json
{
"url": "https://example.com/articles",
@@ -72,16 +86,26 @@ Pagination is restricted to the same scheme and host as the start URL.
"wait_seconds": 2,
"output_format": "json",
"headless": true,
"user_agent": "",
"respect_robots": true,
"rate_limit_seconds": 2,
"screenshot": false
"respect_robots": true
}
```
Iterative planner:
```json
{
"goal": "collect at least 30 product tiles with name, price, link",
"url": "https://example.com/store",
"selectors": {"name": ".tile .name", "price": ".tile .price", "href": ".tile a::attr(href)"},
"min_items": 30,
"max_iterations": 4,
"per_pass_max_pages": 1,
"respect_robots": true
}
```
## Smoke test payload
Use this payload to verify the scraper against the public demo site `https://quotes.toscrape.com/` while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when `screenshot` is `true`.
Use this payload to verify the scraper against the public demo site https://quotes.toscrape.com/ while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when screenshot is true.
```json
{
@@ -105,4 +129,4 @@ Use this payload to verify the scraper against the public demo site `https://quo
## Deployment notes
The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under `/workspace/outputs/seleniumbase-website-scraper/` are durable workspace outputs.
The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under /workspace/outputs/seleniumbase-website-scraper/ are durable workspace outputs.