write README.md
This commit is contained in:
24
README.md
24
README.md
@@ -79,6 +79,30 @@ Pagination is restricted to the same scheme and host as the start URL.
|
||||
}
|
||||
```
|
||||
|
||||
## Smoke test payload
|
||||
|
||||
Use this payload to verify the scraper against the public demo site `https://quotes.toscrape.com/` while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when `screenshot` is `true`.
|
||||
|
||||
```json
|
||||
{
|
||||
"url": "https://quotes.toscrape.com/",
|
||||
"selectors": {
|
||||
"quotes": ".quote",
|
||||
"text": ".text",
|
||||
"author": ".author",
|
||||
"tags": ".tag"
|
||||
},
|
||||
"max_pages": 1,
|
||||
"wait_seconds": 1,
|
||||
"output_format": "json",
|
||||
"headless": true,
|
||||
"user_agent": "",
|
||||
"respect_robots": true,
|
||||
"rate_limit_seconds": 1,
|
||||
"screenshot": false
|
||||
}
|
||||
```
|
||||
|
||||
## Deployment notes
|
||||
|
||||
The agent declares a larger runtime budget because browser work can be CPU/memory intensive. Actual browser execution is performed through the workspace sandbox helper using the SeleniumBase image, so files written under `/workspace/outputs/seleniumbase-website-scraper/` are durable workspace outputs.
|
||||
|
||||
Reference in New Issue
Block a user