[Integrations](https://spiral.ai/integrations) › [Codex](https://spiral.ai/integrations/codex) › [Firecrawl website content](https://spiral.ai/integrations/codex/firecrawl-website-content)

# How to Use Firecrawl website content in Codex

Firecrawl follows a website’s sitemap and links to collect page content as Markdown, HTML, or structured JSON. Use it to research a documentation section or collect material across several pages.

|  |  |
| --- | --- |
| Model | Firecrawl website content |
| Available through | Firecrawl |
| Category | Web |
| Price | $0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each. |
| Returns | json |
| Runs | In the background |
| Setup | About 2 minutes, no API key |

## Prompts to try with Firecrawl website content in Codex

- Use Firecrawl website content to collect https://docs.pydantic.dev/latest/migration/ and relevant linked migration pages. Compare the guidance with this repository’s models and draft a Pydantic migration checklist.
- Use Firecrawl website content to read https://nextjs.org/docs/app/building-your-application/upgrading and its upgrade guides. Compare the returned guidance with package.json and identify changes this project may need.
- Use Firecrawl website content to collect https://docs.astro.build/en/guides/content-collections/ and relevant linked guides. Propose a content collection structure for the Markdown files in this repository.
- Use Firecrawl website content to collect https://fastapi.tiangolo.com/tutorial/security/ and linked security tutorials. Review this project’s authentication code against the returned instructions.
- Use Firecrawl website content to read https://docs.pytest.org/en/stable/how-to/ and relevant linked guides. Recommend fixture and test organization changes based on this repository’s current tests.

## How to set up Firecrawl website content in Codex

Pick the way that suits you. Each takes about two minutes.

### Use Spiral's MCP server

1. **Add the hosted server**

```bash
codex mcp add spiral --url https://api.spiral.ai/mcp
```

2. **Sign in**: Codex opens the browser: sign in to Spiral and click Authorize. No API key. `codex mcp list` then shows Spiral.

```bash
codex mcp login spiral
```

### Use the Spiral skill

1. **Send Codex this message**: Your agent reads Spiral's skill and sets itself up.

```
set up https://spiral.ai/SKILL.md
```

2. **Sign in or paste a key**: Approve Spiral in the browser, or give the agent a key from the console.

### Use the Spiral CLI

1. **Install and sign in**: Installs the Spiral CLI, signs you in and adds its skill to your coding agents.

```bash
npx -y spiral-cli@latest init
```

### Call the Spiral API

1. **Create an API key**: In the console, under API keys.

2. **Run Firecrawl website content**: One request; the result comes back by polling or a webhook.

```bash
curl https://api.spiral.ai/v1/runs \
  -H "Authorization: Bearer $SPIRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "provider": "firecrawl",
    "model": "firecrawl/crawl",
    "input": {
      "url": "<url>"
    }
  }'
```

Then paste one of the prompts above: Codex finds Firecrawl website content on Spiral, shows the price and runs it.

Setup guide in the docs: https://spiral.ai/docs/quickstarts/codex

## Firecrawl website content tools and prices

- [Firecrawl Crawl · Website Pages](https://spiral.ai/tools/firecrawl/firecrawl/crawl) (Firecrawl): $0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each.

## Settings you can ask for

| Setting | Allowed values | Default | Description |
| --- | --- | --- | --- |
| delay |  | — | Delay in seconds between scrapes. This helps respect website rate limits. Setting this forces concurrency to 1. |
| limit |  | 10000 | Maximum number of pages to crawl. Default limit is 10000. |
| prompt |  | — | A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents. |
| sitemap | skip, include, only | include | Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards. If you set it to 'only', the crawler will only crawl URLs from the sitemap (plus the start URL) and will not discover links from HTML. |
| excludePaths |  | — | URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation. |
| includePaths |  | — | URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. Note: the starting URL is also checked against these patterns — if it does not match, the crawl may return 0 pages. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev/blog, only pages under /blog/ will be included in the results, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap. Patterns use Rust regex (RE2-style) syntax: look-around and backreferences are not supported, and a pattern that does not compile is rejected with a 400. Each field accepts at most 1000 patterns of at most 2000 characters each, and includePaths and excludePaths together may contain at most 1000 patterns and 100,000 characters in total. For keyword-style filtering, send one short pattern per term rather than combining terms into a single long alternation. |

## What can you do with Firecrawl website content in Codex?

Turn a public documentation section into an implementation checklist for your project. Your coding agent can compare the collected instructions with repository files and propose changes supported by the returned pages.

### Build a documentation reference

Collect a public documentation section as Markdown and use its page content to answer implementation questions.

### Prepare a migration checklist

Read migration guides across a documentation site and assemble a checklist of changes relevant to your application.

### Compare published product capabilities

Collect public product and documentation pages, then compare supported features using passages from the returned content.

### Review documentation changes

Compare saved crawls of the same documentation section to identify changed instructions and examples.

### Audit your website’s content

Collect your public pages and review their text for outdated descriptions, repeated explanations, and inconsistent terminology.

### Create a structured article inventory

Extract article titles, stated publication dates, and topics from a public blog into structured records for editorial review.

## Overview

- [Firecrawl](https://spiral.ai/providers/firecrawl)
- [All Codex guides](https://spiral.ai/integrations/codex)

## Firecrawl website content in other agents

- [Amp](https://spiral.ai/integrations/amp/firecrawl-website-content)
- [ChatGPT](https://spiral.ai/integrations/chatgpt/firecrawl-website-content)
- [Claude](https://spiral.ai/integrations/claude-desktop/firecrawl-website-content)
- [Claude Code](https://spiral.ai/integrations/claude-code/firecrawl-website-content)
- [Cline](https://spiral.ai/integrations/cline/firecrawl-website-content)
- [Cursor](https://spiral.ai/integrations/cursor/firecrawl-website-content)
- [Devin](https://spiral.ai/integrations/devin/firecrawl-website-content)
- [dots](https://spiral.ai/integrations/dots/firecrawl-website-content)
- [Factory Droid](https://spiral.ai/integrations/factory-droid/firecrawl-website-content)
- [Gemini](https://spiral.ai/integrations/gemini/firecrawl-website-content)
- [Gemini CLI](https://spiral.ai/integrations/gemini-cli/firecrawl-website-content)
- [Google Antigravity](https://spiral.ai/integrations/antigravity/firecrawl-website-content)
- [goose](https://spiral.ai/integrations/goose/firecrawl-website-content)
- [Grok](https://spiral.ai/integrations/grok/firecrawl-website-content)
- [Grok Build](https://spiral.ai/integrations/grok-build/firecrawl-website-content)
- [Hermes](https://spiral.ai/integrations/hermes/firecrawl-website-content)
- [Instinct](https://spiral.ai/integrations/instinct/firecrawl-website-content)
- [Manus](https://spiral.ai/integrations/manus/firecrawl-website-content)
- [Muse](https://spiral.ai/integrations/muse/firecrawl-website-content)
- [Muse Code](https://spiral.ai/integrations/muse-code/firecrawl-website-content)
- [n8n](https://spiral.ai/integrations/n8n/firecrawl-website-content)
- [Notion](https://spiral.ai/integrations/notion/firecrawl-website-content)
- [OpenClaw](https://spiral.ai/integrations/openclaw/firecrawl-website-content)
- [opencode](https://spiral.ai/integrations/opencode/firecrawl-website-content)
- [Perplexity](https://spiral.ai/integrations/perplexity/firecrawl-website-content)
- [Raycast](https://spiral.ai/integrations/raycast/firecrawl-website-content)
- [VS Code](https://spiral.ai/integrations/vscode/firecrawl-website-content)
- [Windsurf](https://spiral.ai/integrations/windsurf/firecrawl-website-content)
- [Zapier](https://spiral.ai/integrations/zapier/firecrawl-website-content)

## More Codex guides

- [Firecrawl website URLs](https://spiral.ai/integrations/codex/firecrawl-website-urls)
- [Google search results](https://spiral.ai/integrations/codex/google-search-results)
- [Firecrawl web search](https://spiral.ai/integrations/codex/firecrawl-web-search)
- [Wikidata item details](https://spiral.ai/integrations/codex/wikidata-item-details)
- [Firecrawl web research](https://spiral.ai/integrations/codex/firecrawl-web-research)
- [Wikidata labels](https://spiral.ai/integrations/codex/wikidata-labels)

## Questions about Firecrawl website content in Codex

**Can Codex use Firecrawl website content?**

Yes. Add Spiral to Codex as an MCP server, and Codex can find and run Firecrawl website content like any built-in tool.

**How much does Firecrawl website content cost in Codex?**

$0.001746 per page crawled ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per page each. You pay from your Spiral balance as you go, with no subscription, and a failed run is not charged. Your Codex plan is separate.

**Do I need a Firecrawl account to use Firecrawl website content in Codex?**

No. Spiral runs Firecrawl website content on its own Firecrawl account. You sign in to Spiral once and never paste a key.

**How long does it take to set up Firecrawl website content in Codex?**

About two minutes: add the hosted server, then sign in.

**Does Codex wait while Firecrawl website content runs?**

Firecrawl website content runs in the background. Codex starts the run, checks on it and gives you the json when it's ready.

**What does Firecrawl collect from a website?**

Firecrawl discovers pages through sitemaps and links, then returns their content as Markdown, HTML, or structured data. JSON extraction uses AI to turn page content into structured records.

**Will a crawl include every page on a site?**

Coverage depends on sitemap and link discovery, robots.txt rules, and the crawl’s scope. Firecrawl documents a default cap of 10,000 pages, and an inclusion rule that rejects the starting URL can produce zero results.

**How fresh is the returned content?**

Firecrawl’s default cache window is two days. A newly requested crawl can therefore include cached page content rather than a new fetch of every page.

**Can a large crawl’s results span multiple responses?**

Yes. Firecrawl provides a next URL for another 10 MB of data when a response is too large or the crawl is still incomplete.

**How long can completed results be retrieved?**

Completed crawl results remain retrievable through Firecrawl’s API for 24 hours. Save the returned content within that window if you need it for later comparisons.

