How to Use Firecrawl batch web scraping in Claude Code
Fetch content from a supplied list of web pages as Markdown, HTML, links, screenshots, or structured JSON. Use it when you already know which URLs to read, such as documentation pages or product pages you want to compare.
Prompts to try with Firecrawl batch web scraping in Claude Code
Copy an example into Claude Code, or ask in your own words.
Use Firecrawl batch web scraping on https://react.dev/reference/react/useState and https://react.dev/reference/react/useEffect. Compare the returned guidance with this project's React components and identify changes worth reviewing.
Use Firecrawl batch web scraping on https://nodejs.org/api/fs.html and https://nodejs.org/api/path.html. Review this project's file handling against the documented behavior and cite the relevant sections.
Use Firecrawl batch web scraping on https://docs.python.org/3/library/json.html and https://docs.python.org/3/library/csv.html. Compare the documented parsing behavior with our import code and flag mismatches.
Use Firecrawl batch web scraping on https://www.postgresql.org/docs/current/sql-select.html and https://www.postgresql.org/docs/current/sql-insert.html. Review the SQL in this repository against the returned documentation.
Use Firecrawl batch web scraping on https://react.dev/reference/react/useState and https://react.dev/reference/react/useReducer. Create a short reference in the project's docs explaining which existing components could benefit from a reducer.
How to set up Firecrawl batch web scraping in Claude Code
Pick the way that suits you. Each takes about two minutes.
Use Spiral's MCP server
- Add the hosted server
This adds Spiral to the current project. Add --scope user to use it in every project.
claude mcp add --transport http spiral https://api.spiral.ai/mcp
- Sign in
Start claude, type /mcp, choose spiral and Authenticate. Sign in to Spiral in the browser and click Authorize. No API key.
Use the Spiral skill
- Send Claude Code this message
Your agent reads Spiral's skill and sets itself up.
set up https://spiral.ai/SKILL.md
- Sign in or paste a key
Approve Spiral in the browser, or give the agent a key from the console.
Use the Spiral CLI
- Install and sign in
Installs the Spiral CLI, signs you in and adds its skill to your coding agents.
npx -y spiral-cli@latest init
Call the Spiral API
- Create an API key
In the console, under API keys.
- Run Firecrawl batch web scraping
One request; the result comes back by polling or a webhook.
curl https://api.spiral.ai/v1/runs \ -H "Authorization: Bearer $SPIRAL_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "provider": "firecrawl", "model": "firecrawl/batch-scrape", "input": { "urls": [ "<urls>" ] } }'
Then paste one of the prompts above: Claude Code finds Firecrawl batch web scraping on Spiral, shows the price and runs it.
Firecrawl batch web scraping providers and prices
Spiral runs Firecrawl batch web scraping through Firecrawl. Open a card for today's price, with every fee included.
$0.001746 per URL ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per URL each.
Settings you can ask for
Ask Claude Code for any of these in your prompt.
| Setting | Allowed values | Default | Description |
|---|---|---|---|
| Proxy | basic, enhanced, or auto | auto | Specifies the type of proxy to use.
|
| Max age | a whole number | 172800000 | Returns a cached version of the page if it is younger than this age in milliseconds. If a cached version of the page is older than this value, the page will be scraped. If you do not need extremely fresh data, enabling this can speed up your scrapes by 500%. Defaults to 2 days. |
| Min age | a whole number | When set, the request only checks the cache and never triggers a fresh scrape. The value is in milliseconds and specifies the minimum age the cached data must be. If matching cached data exists, it is returned instantly. If no cached data is found, a 404 with error code SCRAPE_NO_CACHED_DATA is returned. Set to 1 to accept any cached data regardless of age. | |
| Mobile | on or off | off | Set to true if you want to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots. |
| Timeout | 1000 to 300000 | 60000 | Timeout in milliseconds for the request. Minimum is 1000 (1 second). Default is 60000 (60 seconds). Maximum is 300000 (300 seconds). |
| Wait for | a whole number up to 60000 | 0 | Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load. This waiting time is in addition to Firecrawl's smart wait feature. |
| Block ads | on or off | on | Enables ad-blocking and cookie popup blocking. |
| Lockdown | on or off | off | If true, serves the request from Firecrawl's cache only and never makes an outbound request to the target URL. Designed for compliance-constrained or air-gapped environments where the scrape request itself could leak sensitive information. On cache miss, returns a 404 with error code SCRAPE_LOCKDOWN_CACHE_MISS (the URL is never logged on miss). Lockdown requests are treated as zero data retention. Default maxAge is extended to 2 years so existing cached pages remain eligible. Billed at 5 credits on hit, 1 credit on cache miss. |
| Store in cache | on or off | on | If true, the page will be stored in the Firecrawl index and cache. Setting this to false is useful if your scraping activity may have data protection concerns. Using some parameters associated with sensitive scraping (e.g. actions, headers) will force this parameter to be false. |
| Max concurrency | a whole number | Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this batch scrape. If not specified, the batch scrape adheres to your team's concurrency limit. | |
| Only main content | on or off | on | Only return the main content of the page excluding headers, navs, footers, etc. This is a deterministic HTML-level filter applied before markdown is generated; no LLM is involved. |
| Only clean content | on or off | off | Beta. Run an additional LLM-based pass over the generated markdown to remove residual boilerplate that |
| Ignore invalid URLs | on or off | on | If invalid URLs are specified in the urls array, they will be ignored. Instead of them failing the entire request, a batch scrape using the remaining valid URLs will be created, and the invalid URLs will be returned in the invalidURLs field of the response. |
| Remove base64 images | on or off | on | Removes all base 64 images from the markdown output, which may be overwhelmingly long. This does not affect html or rawHtml formats. The image's alt text remains in the output, but the URL is replaced with a placeholder. |
| Skip tls verification | on or off | off | Skip TLS certificate verification when making requests. |
| Check prompt injection | on or off | off | When enabled, scans the page content for prompt injection attempts before any LLM-backed format (such as json, summary or question) runs. Works with any format except rawBase64. If an injection is detected, the request fails with a 403 and error code SCRAPE_PROMPT_INJECTION_DETECTED. Adds 4 credits when the check runs. If the check cannot scan all of the content, the scrape continues with a warning and the check is not billed. |
Every API field of Firecrawl Batch Scrape · Many Pages is on its tool page: open its card above.
What can you do with Firecrawl batch web scraping in Claude Code?
Build a documentation reference that matches the code you are changing. Firecrawl supplies content from selected URLs so a coding agent can compare published guidance with the project's implementation.
Build a documentation reference
Collect a selected set of documentation URLs as page text, then organize the returned content into a searchable reference with links to the original pages.
Compare public product descriptions
Read several public product pages and build a table of stated features, supported platforms, and published specifications.
Prepare a migration review
Fetch the documentation pages relevant to a planned code change and compare their examples and requirements with the project's implementation.
Create a release digest
Fetch selected release-note pages on a schedule and compare their text with saved snapshots to summarize newly published changes.
Review public policy changes
Collect a fixed list of public terms or policy pages and compare successive snapshots, highlighting the passages that changed.
Audit selected website pages
Inspect returned page text and links across an approved URL list to find inconsistent descriptions, outdated references, or unexpected navigation links.
Compare page presentation
Collect screenshots and page text for selected pages, then review how their visible presentation matches their stated content.
Firecrawl batch web scraping success rate and speed on Spiral
Success and speed over the last 24 hours, across every run of these tools on Spiral.
| Tool | Success | Median time | Runs |
|---|---|---|---|
| Firecrawl Batch Scrape · Many Pages | 95.2% | 14.8s | 63 |
More on Firecrawl batch web scraping and Claude Code
Overview
Firecrawl batch web scraping in other agents
Questions about Firecrawl batch web scraping in Claude Code
Can Claude Code use Firecrawl batch web scraping?
Yes. Add Spiral to Claude Code as an MCP server, and Claude Code can find and run Firecrawl batch web scraping like any built-in tool.
How much does Firecrawl batch web scraping cost in Claude Code?
$0.001746 per URL ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per URL each. You pay from your Spiral balance as you go, with no subscription, and a failed run is not charged. Your Claude Code plan is separate.
Do I need a Firecrawl account to use Firecrawl batch web scraping in Claude Code?
No. Spiral runs Firecrawl batch web scraping on its own Firecrawl account. You sign in to Spiral once and never paste a key.
How long does it take to set up Firecrawl batch web scraping in Claude Code?
About two minutes: add the hosted server, then sign in.
Does Claude Code wait while Firecrawl batch web scraping runs?
Firecrawl batch web scraping runs in the background. Claude Code starts the run, checks on it and gives you the json when it's ready.
Does Firecrawl batch scraping collect an entire website?
It processes an explicit list of URLs. Its coverage follows that list, so include each page you want represented; some websites are also unsupported.
How fresh is the returned page content?
Firecrawl's default cache freshness window is two days. A new request can therefore return a recent cached copy rather than newly fetched website content.
Can one result response contain the whole batch?
Large results require pagination: Firecrawl provides a continuation URL for the next 10 MB of data. The response also reports how many pages were successfully scraped.
How long can I retrieve completed results?
Batch results are available through the API for 24 hours after completion. Save the returned content within that window if you need it for later comparisons or indexing.
Can structured JSON include hidden HTML attributes?
JSON extraction works from the page's Markdown conversion, which preserves visible text. HTML attributes are unavailable to that extraction, so use original HTML when those values matter.
Start using Firecrawl batch web scraping in Claude Code
One Spiral account for every tool your agent needs. No subscription; you pay per run.
Paste this into Claude Code:
set up https://spiral.ai/SKILL.md


