seleniumbase-website-scraper

Private A2A agent for compliant, authorized website scraping with SeleniumBase/browser automation where available.

Safety policy

Use this agent only for authorized scraping of public or user-owned sites. The agent must not bypass, solve, defeat, or evade CAPTCHAs, anti-bot protections, login gates, paywalls, robots.txt, ToS restrictions, or access controls. If a CAPTCHA, anti-bot challenge, login wall, paywall, or access-control indicator is detected, the run stops and returns a manual-intervention/blocked status.

The agent does not use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential injection, or access-control circumvention.

Skill

`scrape_website`

Inputs:

url — absolute http or https URL.
selectors — mapping of output field names to CSS selectors.
max_pages — maximum same-origin pages to visit.
wait_seconds — wait after page open before capture/extraction.
output_format — json or csv.
headless — toggle browser headless mode.
user_agent — optional custom single-line user agent.
respect_robots — when true, checks robots.txt before fetch and pagination.
rate_limit_seconds — polite delay between pagination requests.
screenshot — save PNG screenshots for visited pages.

Outputs:

status — ok, invalid_input, blocked_by_robots_txt, manual_intervention_required, browser_runtime_unavailable, or error.
items — extracted records.
saved_paths — output files under outputs/seleniumbase-website-scraper/.
warnings — safety, robots, selector, browser, or runtime warnings.
screenshots — screenshot file paths when requested.
pages_visited and html_captures.

Selector syntax

Default extraction returns visible text:

{"title": "h1", "summary": ".article-summary"}

Supported suffixes:

::text — visible text.
::html — inner HTML.
::attr(name) — attribute extraction, e.g. a.card::attr(href).

Special pagination selector field names are not emitted as data fields and are used to find the next page:

next
_next
__next__
pagination_next
next_page

Pagination is restricted to the same scheme and host as the start URL.

Example payload

{
  "url": "https://example.com/articles",
  "selectors": {
    "headline": "h2.article-title",
    "links": "a.article-link::attr(href)",
    "next": "a[rel='next']::attr(href)"
  },
  "max_pages": 3,
  "wait_seconds": 2,
  "output_format": "json",
  "headless": true,
  "user_agent": "",
  "respect_robots": true,
  "rate_limit_seconds": 2,
  "screenshot": false
}

Deployment notes

The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under /workspace/outputs/seleniumbase-website-scraper/ are durable workspace outputs.

2.9 KiB Raw Blame History