46 lines
2.2 KiB
Markdown
46 lines
2.2 KiB
Markdown
---
|
|
name: compliant-browser-scraping
|
|
description: "Safety and workflow guidance for authorized public or user-owned browser scraping with SeleniumBase. Use for website scraping runs, selector extraction, pagination, robots.txt checks, challenge detection, and artifact reporting."
|
|
---
|
|
# compliant-browser-scraping
|
|
|
|
# Compliant Browser Scraping
|
|
|
|
This agent is for authorized scraping of public or user-owned websites only.
|
|
|
|
## Mandatory safety rules
|
|
|
|
- Do not bypass, solve, defeat, evade, or work around CAPTCHAs, anti-bot protections, login gates, paywalls, robots.txt restrictions, ToS restrictions, IP blocks, access controls, or technical protection measures.
|
|
- If a CAPTCHA, anti-bot challenge, login wall, paywall, access denial, or equivalent challenge is detected, stop the run and return a clear `manual_intervention_required` or `blocked` status with warnings.
|
|
- Do not use stealth/undetected-browser modes, CAPTCHA solvers, proxy rotation, credential stuffing, cookie injection, or other circumvention techniques.
|
|
- Respect robots.txt when requested by the caller.
|
|
- Paginate only within the same scheme+host as the starting URL unless the caller starts a new scrape explicitly for another URL.
|
|
- Apply polite rate limiting between page visits.
|
|
|
|
## Selector mapping conventions
|
|
|
|
The public `selectors` input is a mapping from output field names to CSS selectors. Examples:
|
|
|
|
```json
|
|
{
|
|
"title": "h1",
|
|
"price": ".price",
|
|
"links": "a.result::attr(href)",
|
|
"next": "a[rel='next']::attr(href)"
|
|
}
|
|
```
|
|
|
|
Special selector names `next`, `_next`, `__next__`, `pagination_next`, or `next_page` are interpreted as pagination selectors and are not emitted as data fields.
|
|
|
|
Selector suffixes:
|
|
|
|
- `::text` extracts visible text (default behavior for element selectors).
|
|
- `::html` extracts innerHTML.
|
|
- `::attr(name)` extracts an attribute.
|
|
|
|
When multiple elements match a field, return a list. When one element matches, return a string. When nothing matches, return null.
|
|
|
|
## Output artifacts
|
|
|
|
Save machine-readable output under `outputs/seleniumbase-website-scraper/` as JSON or CSV. If screenshots are requested, save PNG screenshots in the same directory. Include warnings and status in the response.
|