a2a-source-edit: write README.md
This commit is contained in:
a2a-cloud
145
README.md
Normal file
145
README.md
Normal file
@@ -0,0 +1,145 @@
|
|||||||
|
# seleniumbase-website-scraper
|
||||||
|
|
||||||
|
Private A2A agent for compliant, authorized website scraping with SeleniumBase/browser automation where available.
|
||||||
|
|
||||||
|
## Safety policy
|
||||||
|
|
||||||
|
Use this agent only for authorized scraping of public or user-owned sites. The agent must not bypass, solve, defeat, or evade CAPTCHAs, anti-bot protections, login gates, paywalls, `robots.txt`, ToS restrictions, or access controls. If a CAPTCHA, anti-bot challenge, login wall, paywall, or access-control indicator is detected, the run stops and returns a manual-intervention/blocked status.
|
||||||
|
|
||||||
|
The agent does not use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention.
|
||||||
|
|
||||||
|
## Skills
|
||||||
|
|
||||||
|
### scrape_website
|
||||||
|
|
||||||
|
Inputs:
|
||||||
|
- url — absolute http or https URL.
|
||||||
|
- selectors — mapping of output field names to CSS selectors.
|
||||||
|
- max_pages — maximum same-origin pages to visit.
|
||||||
|
- wait_seconds — wait after page open before capture/extraction.
|
||||||
|
- output_format — json or csv.
|
||||||
|
- headless — toggle browser headless mode.
|
||||||
|
- user_agent — optional custom single-line user agent.
|
||||||
|
- respect_robots — when true, checks robots.txt before fetch and pagination.
|
||||||
|
- rate_limit_seconds — polite delay between pagination requests.
|
||||||
|
- screenshot — save PNG screenshots for visited pages.
|
||||||
|
|
||||||
|
Outputs:
|
||||||
|
- status — ok, invalid_input, blocked_by_robots_txt, manual_intervention_required, browser_runtime_unavailable, or error.
|
||||||
|
- items — extracted records.
|
||||||
|
- saved_paths — output files under outputs/seleniumbase-website-scraper/.
|
||||||
|
- warnings — safety, robots, selector, browser, or runtime warnings.
|
||||||
|
- screenshots — screenshot file paths when requested.
|
||||||
|
- pages_visited and html_captures.
|
||||||
|
|
||||||
|
### iterate_scrape_goal (new)
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- start_url, allowed_domains (default same-origin), seed_selectors, max_steps, max_pages,
|
||||||
|
max_runtime_seconds, wait_seconds, rate_limit_seconds, headless, user_agent,
|
||||||
|
respect_robots (default true), output_format (json/csv), screenshot.
|
||||||
|
|
||||||
|
Returns fields include: {status, items, pages_visited, html_captures, screenshots, saved_paths, warnings, steps, goal_summary}.
|
||||||
|
|
||||||
|
An LLM-powered planner that iterates scrapes until a goal is satisfied or a safety/stop condition occurs.
|
||||||
|
Inputs:
|
||||||
|
- goal — natural-language target (e.g., "Collect quotes text and authors from the first 2 pages").
|
||||||
|
- start_url — start page (absolute http/https). Same-origin only for all iterations unless allowed_domains is provided.
|
||||||
|
- seed_selectors — optional initial CSS selector map (defaults to {"title": "title"}).
|
||||||
|
- max_steps — planner steps (hard-capped at 3 improvements).
|
||||||
|
- max_pages — pages per scrape pass (default 1, capped by agent config).
|
||||||
|
- max_runtime_seconds — total runtime budget across steps.
|
||||||
|
- wait_seconds, output_format, headless, user_agent, respect_robots, rate_limit_seconds, screenshot — mirror scrape_website.
|
||||||
|
- allowed_domains — optional whitelist; else restricted to same-origin.
|
||||||
|
|
||||||
|
Outputs:
|
||||||
|
- status — ok | no_items | blocked_by_robots_txt | manual_intervention_required | browser_runtime_unavailable | error
|
||||||
|
- items — accumulated unique items across passes
|
||||||
|
- pages_visited — union of pages visited across passes
|
||||||
|
- html_captures — captures from passes
|
||||||
|
- screenshots — when requested
|
||||||
|
- saved_paths — includes an iter-plan manifest under outputs/seleniumbase-website-scraper/
|
||||||
|
- warnings — including planner parsing issues, safety stops, or budget stops
|
||||||
|
- iterations — number of planner decisions taken
|
||||||
|
- decisions — array of {iteration, action, reason, confidence, url?, selectors?}
|
||||||
|
|
||||||
|
Behavior and guarantees:
|
||||||
|
- Uses platform-provided ctx.llm with strict system prompt that forbids any evasion of CAPTCHAs, anti-bot, logins, paywalls, or robots.txt.
|
||||||
|
- Planner can only propose same-origin URLs; cross-origin plans are rejected and end the loop.
|
||||||
|
- Each pass runs the deterministic scrape_website skill; stops on any challenge or robots block.
|
||||||
|
- Produces a compact decisions trail and saves a small manifest under outputs/seleniumbase-website-scraper/.
|
||||||
|
|
||||||
|
## Selector syntax
|
||||||
|
|
||||||
|
Default extraction returns visible text:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"title": "h1", "summary": ".article-summary"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Supported suffixes:
|
||||||
|
- ::text — visible text.
|
||||||
|
- ::html — inner HTML.
|
||||||
|
- ::attr(name) — attribute extraction, e.g. a.card::attr(href).
|
||||||
|
|
||||||
|
Special pagination selector field names are not emitted as data fields and are used to find the next page:
|
||||||
|
- next
|
||||||
|
- _next
|
||||||
|
- __next__
|
||||||
|
- pagination_next
|
||||||
|
- next_page
|
||||||
|
|
||||||
|
Pagination is restricted to the same scheme and host as the start URL.
|
||||||
|
|
||||||
|
## Example payloads
|
||||||
|
|
||||||
|
Simple one-pass scrape:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"url": "https://example.com/articles",
|
||||||
|
"selectors": {
|
||||||
|
"headline": "h2.article-title",
|
||||||
|
"links": "a.article-link::attr(href)",
|
||||||
|
"next": "a[rel='next']::attr(href)"
|
||||||
|
},
|
||||||
|
"max_pages": 3,
|
||||||
|
"wait_seconds": 2,
|
||||||
|
"output_format": "json",
|
||||||
|
"headless": true,
|
||||||
|
"respect_robots": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Iterative planner:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"goal": "collect at least 30 product tiles with name, price, link",
|
||||||
|
"url": "https://example.com/store",
|
||||||
|
"selectors": {"name": ".tile .name", "price": ".tile .price", "href": ".tile a::attr(href)"},
|
||||||
|
"min_items": 30,
|
||||||
|
"max_iterations": 3,
|
||||||
|
"per_pass_max_pages": 1,
|
||||||
|
"respect_robots": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Smoke test: quotes.toscrape.com
|
||||||
|
|
||||||
|
Goal: "Collect quotes text and authors from the first 2 pages"
|
||||||
|
- Expected: status=ok, >= 1 item, <= 2 iterations (often 1–2), a saved JSON manifest, no screenshots when disabled.
|
||||||
|
- Example call (schematic):
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"goal": "Collect quotes text and authors from the first 2 pages",
|
||||||
|
"url": "https://quotes.toscrape.com/",
|
||||||
|
"selectors": {"text": ".quote .text", "author": ".quote .author", "pagination_next": "li.next a::attr(href)"},
|
||||||
|
"min_items": 2,
|
||||||
|
"max_iterations": 2,
|
||||||
|
"per_pass_max_pages": 2,
|
||||||
|
"screenshot": false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment notes
|
||||||
|
|
||||||
|
The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under /workspace/outputs/seleniumbase-website-scraper/ are durable workspace outputs.
|
||||||
Reference in New Issue
Block a user