Spiral gives your agent every tool with one account. Browse the tools →

FirecrawlWebAvailable now

Firecrawl Crawl · Website Pages

firecrawl/crawl

Search and scrape the web with Firecrawl Crawl · Website Pages through one Spiral API key. Crawls a website from a starting URL within the paths and depth you set, returning each page as markdown, HTML or JSON. It runs in the background and returns structured JSON. Pay as you go, with no subscription.

Pricing

$0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each.

Limits

Runs
In the background
Returns
json
Longest queue wait
1 hour
Run deadline
1 hour
Longest run, end to end
225 minutes
Largest inline input
1 MB
Largest input file
2 GB
Largest output
5 GB

Input

FieldKindDefaultDescription
url requiredtext—

The base URL to start crawling from

delaynumber—

Delay in seconds between scrapes. This helps respect website rate limits. Setting this forces concurrency to 1.

limitnumber (whole)10000

Maximum number of pages to crawl. Default limit is 10000.

prompttext—

A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents.

sitemapone of
skip, include, only
include

Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards. If you set it to 'only', the crawler will only crawl URLs from the sitemap (plus the start URL) and will not discover links from HTML.

excludePathslist—

URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation.

includePathslist—

URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. Note: the starting URL is also checked against these patterns — if it does not match, the crawl may return 0 pages. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev/blog, only pages under /blog/ will be included in the results, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation.

scrapeOptionsJSON—
maxConcurrencynumber (whole)—

Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this crawl. If not specified, the crawl adheres to your team's concurrency limit.

regexOnFullURLtrue or falsefalse

When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters) instead of just the URL pathname. Useful when you need to filter URLs based on query strings.

allowSubdomainstrue or falsefalse

Allows the crawler to follow links to subdomains of the main domain.

ignoreRobotsTxttrue or falsefalse

Ignore the website's robots.txt rules. Enterprise only — contact [email protected] to enable.

robotsUserAgenttext—

Custom User-Agent string for robots.txt evaluation. When set, robots.txt is fetched with this User-Agent and allow/disallow rules are matched against it instead of the default. Enterprise only — contact [email protected] to enable.

crawlEntireDomaintrue or falsefalse

Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.

false: Only crawls deeper (child) URLs.

→ e.g. /features/feature-1 → /features/feature-1/tips ✅

→ Won't follow /pricing or / ❌

true: Crawls any internal links, including siblings and parents.

→ e.g. /features/feature-1 → /pricing, /, etc. ✅

Use true for broader internal coverage beyond nested paths.

maxDiscoveryDepthnumber (whole)—

Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set sitemap: 'skip', you will only crawl the entered URL and all URLs that are linked on that page.

allowExternalLinkstrue or falsefalse

Allows the crawler to follow links to external websites. External links are followed one hop (the links found on those external pages are not crawled). Links pointing to an external site's homepage (a root URL with no path) are skipped and reported in Get Crawl Errors with the code EXTERNAL_LINK; redirects to an external homepage are skipped for the same reason.

ignoreQueryParameterstrue or falsefalse

Do not re-scrape the same path with different (or none) query parameters

Same request in code

curl https://api.spiral.ai/v1/runs \
  -H "Authorization: Bearer $SPIRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "provider": "firecrawl",
    "model": "firecrawl/crawl",
    "input": {
      "url": "<url>"
    }
  }'

Frequently asked questions

How much does Firecrawl Crawl · Website Pages cost?

$0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each. You pay as you go from one Spiral balance, with no subscription, and a failed run is not charged.

How do I use Firecrawl Crawl · Website Pages through an API?

Send `POST /v1/runs` with `"provider": "firecrawl"`, `"model": "firecrawl/crawl"` and the input fields on this page, using one Spiral API key. It runs in the background: poll the run, or name a webhook to be told when it finishes.

Do I need an account with Firecrawl to use Firecrawl Crawl · Website Pages?

No. Spiral runs it on its own Firecrawl account and charges your Spiral balance.

Can my AI agent use Firecrawl Crawl · Website Pages?

Yes. Agents find and run it through Spiral's MCP server, CLI or agent skill, with the same API key.