FirecrawlWeb crawlComing soon

firecrawl/crawl

Name: firecrawl/crawl
Brand: Firecrawl

firecrawl/crawl is a web crawl tool from Firecrawl that runs in the background and returns structured JSON. Coming soon: listed with its price, not yet open to runs.

This tool is listed but not yet available to run.

Priced by usage

Pricing

Up to $32.50 per run at the default inputs. The final charge comes from usage.

Limits

Runs: In the background
Returns: json
Longest queue wait: 3600s
Run deadline: 1800s
Largest inline input: 1024 KB
Largest input file: 500 MB
Largest output: 4768 MB
Outputs per run: 16

Input

Field	Kind	Default	Description
`url` required	text	—	The base URL to start crawling from
`delay`	number	—	Delay in seconds between scrapes. This helps respect website rate limits. Setting this forces concurrency to 1.
`limit`	number (whole)	10000	Maximum number of pages to crawl. Default limit is 10000.
`prompt`	text	—	A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents.
`sitemap`	one of skip, include, only	include	Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards. If you set it to 'only', the crawler will only crawl URLs from the sitemap (plus the start URL) and will not discover links from HTML.
`excludePaths`	list	—	URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation.
`includePaths`	list	—	URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. Note: the starting URL is also checked against these patterns — if it does not match, the crawl may return 0 pages. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev/blog, only pages under /blog/ will be included in the results, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation.
`scrapeOptions`	JSON	—
`maxConcurrency`	number (whole)	—	Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this crawl. If not specified, the crawl adheres to your team's concurrency limit.
`regexOnFullURL`	true or false	false	When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters) instead of just the URL pathname. Useful when you need to filter URLs based on query strings.
`allowSubdomains`	true or false	false	Allows the crawler to follow links to subdomains of the main domain.
`ignoreRobotsTxt`	true or false	false	Ignore the website's robots.txt rules. Enterprise only — contact [email protected] to enable.
`robotsUserAgent`	text	—	Custom User-Agent string for robots.txt evaluation. When set, robots.txt is fetched with this User-Agent and allow/disallow rules are matched against it instead of the default. Enterprise only — contact [email protected] to enable.
`crawlEntireDomain`	true or false	false	Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. false: Only crawls deeper (child) URLs. → e.g. /features/feature-1 → /features/feature-1/tips ✅ → Won't follow /pricing or / ❌ true: Crawls any internal links, including siblings and parents. → e.g. /features/feature-1 → /pricing, /, etc. ✅ Use true for broader internal coverage beyond nested paths.
`maxDiscoveryDepth`	number (whole)	—	Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set `sitemap: 'skip'`, you will only crawl the entered URL and all URLs that are linked on that page.
`zeroDataRetention`	true or false	false	If true, this will enable zero data retention for this crawl. To enable this feature, please contact [email protected]
`allowExternalLinks`	true or false	false	Allows the crawler to follow links to external websites. External links are followed one hop (the links found on those external pages are not crawled). Links pointing to an external site's homepage (a root URL with no path) are skipped and reported in Get Crawl Errors with the code EXTERNAL_LINK; redirects to an external homepage are skipped for the same reason.
`ignoreQueryParameters`	true or false	false	Do not re-scrape the same path with different (or none) query parameters

Same request in code

curl https://api.spiral.ai/v1/runs \
  -H "Authorization: Bearer $SPIRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "provider": "firecrawl",
    "model": "firecrawl/crawl",
    "input": {
      "url": "<url>"
    }
  }'