Firecrawl Crawl · Website Pages
firecrawl/crawlSearch and scrape the web with Firecrawl Crawl · Website Pages through one Spiral API key. Crawls a website from a starting URL within the paths and depth you set, returning each page as markdown, HTML or JSON. It runs in the background and returns structured JSON. Pay as you go, with no subscription.
Pricing
$0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each.
Limits
- Runs
- In the background
- Returns
- json
- Longest queue wait
- 1 hour
- Run deadline
- 1 hour
- Longest run, end to end
- 225 minutes
- Largest inline input
- 1 MB
- Largest input file
- 2 GB
- Largest output
- 5 GB
Input
| Field | Kind | Default | Description |
|---|---|---|---|
url required | text | — | The base URL to start crawling from |
delay | number | — | Delay in seconds between scrapes. This helps respect website rate limits. Setting this forces concurrency to 1. |
limit | number (whole) | 10000 | Maximum number of pages to crawl. Default limit is 10000. |
prompt | text | — | A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents. |
sitemap | one of skip, include, only | include | Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards. If you set it to 'only', the crawler will only crawl URLs from the sitemap (plus the start URL) and will not discover links from HTML. |
excludePaths | list | — | URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation. |
includePaths | list | — | URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. Note: the starting URL is also checked against these patterns — if it does not match, the crawl may return 0 pages. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev/blog, only pages under /blog/ will be included in the results, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation. |
scrapeOptions | JSON | — | |
maxConcurrency | number (whole) | — | Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this crawl. If not specified, the crawl adheres to your team's concurrency limit. |
regexOnFullURL | true or false | false | When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters) instead of just the URL pathname. Useful when you need to filter URLs based on query strings. |
allowSubdomains | true or false | false | Allows the crawler to follow links to subdomains of the main domain. |
ignoreRobotsTxt | true or false | false | Ignore the website's robots.txt rules. Enterprise only — contact [email protected] to enable. |
robotsUserAgent | text | — | Custom User-Agent string for robots.txt evaluation. When set, robots.txt is fetched with this User-Agent and allow/disallow rules are matched against it instead of the default. Enterprise only — contact [email protected] to enable. |
crawlEntireDomain | true or false | false | Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. false: Only crawls deeper (child) URLs. → e.g. /features/feature-1 → /features/feature-1/tips ✅ → Won't follow /pricing or / ❌ true: Crawls any internal links, including siblings and parents. → e.g. /features/feature-1 → /pricing, /, etc. ✅ Use true for broader internal coverage beyond nested paths. |
maxDiscoveryDepth | number (whole) | — | Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set |
allowExternalLinks | true or false | false | Allows the crawler to follow links to external websites. External links are followed one hop (the links found on those external pages are not crawled). Links pointing to an external site's homepage (a root URL with no path) are skipped and reported in Get Crawl Errors with the code EXTERNAL_LINK; redirects to an external homepage are skipped for the same reason. |
ignoreQueryParameters | true or false | false | Do not re-scrape the same path with different (or none) query parameters |
Same request in code
curl https://api.spiral.ai/v1/runs \
-H "Authorization: Bearer $SPIRAL_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{
"provider": "firecrawl",
"model": "firecrawl/crawl",
"input": {
"url": "<url>"
}
}'Frequently asked questions
How much does Firecrawl Crawl · Website Pages cost?
$0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each. You pay as you go from one Spiral balance, with no subscription, and a failed run is not charged.
How do I use Firecrawl Crawl · Website Pages through an API?
Send `POST /v1/runs` with `"provider": "firecrawl"`, `"model": "firecrawl/crawl"` and the input fields on this page, using one Spiral API key. It runs in the background: poll the run, or name a webhook to be told when it finishes.
Do I need an account with Firecrawl to use Firecrawl Crawl · Website Pages?
No. Spiral runs it on its own Firecrawl account and charges your Spiral balance.
Can my AI agent use Firecrawl Crawl · Website Pages?
Yes. Agents find and run it through Spiral's MCP server, CLI or agent skill, with the same API key.