write README.md

README.md

@@ -35,17 +35,29 @@ Outputs:
 ### iterate_scrape_goal (new)
 
 An LLM-powered planner that iterates scrapes until a goal is satisfied or a safety/stop condition occurs.
-- goal — natural-language target (e.g., "collect at least 50 product cards with title, price, href").
+Inputs:
+- goal — natural-language target (e.g., "Collect quotes text and authors from the first 2 pages").
 - url — start page (absolute http/https). Same-origin only for all iterations.
 - selectors — optional initial CSS selector map (defaults to {"title": "title"}).
 - min_items — stop when this many items have been collected.
-- max_iterations — planner budget (default 5).
+- max_iterations — planner budget (default 5, hard-capped at 3 improvements per runtime notes). 
 - per_pass_max_pages — pages per scrape pass (default 1, capped by agent config).
-- Other browser and safety flags mirror scrape_website.
+- wait_seconds, output_format, headless, user_agent, respect_robots, rate_limit_seconds, screenshot — mirror scrape_website.
+
+Outputs:
+- status — ok | no_items | blocked_by_robots_txt | manual_intervention_required | browser_runtime_unavailable | error
+- items — accumulated unique items across passes
+- pages_visited — union of pages visited across passes
+- html_captures — captures from passes
+- screenshots — when requested
+- saved_paths — includes an iter-plan manifest under outputs/seleniumbase-website-scraper/
+- warnings — including planner parsing issues, safety stops, or budget stops
+- iterations — number of planner decisions taken
+- decisions — array of {iteration, action, reason, confidence, url?, selectors?}
 
 Behavior and guarantees:
 - Uses platform-provided ctx.llm with strict system prompt that forbids any evasion of CAPTCHAs, anti-bot, logins, paywalls, or robots.txt.
-- Planner can only propose same-origin URLs and CSS selectors. Cross-origin plans are rejected.
+- Planner can only propose same-origin URLs; cross-origin plans are rejected and end the loop.
 - Each pass runs the deterministic scrape_website skill; stops on any challenge or robots block.
 - Produces a compact decisions trail and saves a small manifest under outputs/seleniumbase-website-scraper/.
 
@@ -97,32 +109,25 @@ Iterative planner:
   "url": "https://example.com/store",
   "selectors": {"name": ".tile .name", "price": ".tile .price", "href": ".tile a::attr(href)"},
   "min_items": 30,
-  "max_iterations": 4,
+  "max_iterations": 3,
   "per_pass_max_pages": 1,
   "respect_robots": true
 }
 ```
 
-## Smoke test payload
-
-Use this payload to verify the scraper against the public demo site https://quotes.toscrape.com/ while respecting robots.txt. It should return parseable JSON, quote items, a saved JSON file, an HTML capture path, and screenshot paths only when screenshot is true.
+## Smoke test: quotes.toscrape.com
 
+Goal: "Collect quotes text and authors from the first 2 pages"
+- Expected: status=ok, >= 1 item, <= 2 iterations (often 1–2), a saved JSON manifest, no screenshots when disabled.
+- Example call (schematic):
 ```json
 {
+  "goal": "Collect quotes text and authors from the first 2 pages",
   "url": "https://quotes.toscrape.com/",
-  "selectors": {
-    "quotes": ".quote",
-    "text": ".text",
-    "author": ".author",
-    "tags": ".tag"
-  },
-  "max_pages": 1,
-  "wait_seconds": 1,
-  "output_format": "json",
-  "headless": true,
-  "user_agent": "",
-  "respect_robots": true,
-  "rate_limit_seconds": 1,
+  "selectors": {"text": ".quote .text", "author": ".quote .author", "pagination_next": "li.next a::attr(href)"},
+  "min_items": 2,
+  "max_iterations": 2,
+  "per_pass_max_pages": 2,
   "screenshot": false
 }
 ```