[Integrations](https://spiral.ai/integrations) › [Hermes](https://spiral.ai/integrations/hermes) › [Firecrawl batch web scraping](https://spiral.ai/integrations/hermes/firecrawl-batch-web-scraping)

# How to Use Firecrawl batch web scraping in Hermes

Fetch content from a supplied list of web pages as Markdown, HTML, links, screenshots, or structured JSON. Use it when you already know which URLs to read, such as documentation pages or product pages you want to compare.

|  |  |
| --- | --- |
| Model | Firecrawl batch web scraping |
| Available through | Firecrawl |
| Category | Web |
| Price | $0.001746 per URL ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per URL each. |
| Returns | json |
| Runs | In the background |
| Setup | About 2 minutes, no API key |

## Prompts to try with Firecrawl batch web scraping in Hermes

- Every morning, use Firecrawl batch web scraping on https://www.python.org/downloads/ and https://nodejs.org/en/download. Request fresh page content and report changes to the visible release names and version numbers since the previous run.
- Every Monday, use Firecrawl batch web scraping on https://react.dev/reference/react/useState and https://react.dev/reference/react/useEffect. Request fresh content and summarize changed guidance or examples compared with the previous run.
- Every week, use Firecrawl batch web scraping on https://www.mozilla.org/en-US/privacy/ and https://www.mozilla.org/en-US/about/. Request fresh content and show any changed passages with their source URLs.
- Every Friday, use Firecrawl batch web scraping on https://docs.python.org/3/library/json.html and https://docs.python.org/3/library/csv.html. Request fresh content and report changes to documented behavior since the previous run.
- Every morning, use Firecrawl batch web scraping on https://nodejs.org/en/blog/release and https://www.python.org/downloads/. Request fresh content and report newly visible release headings and links compared with the previous run.

## How to set up Firecrawl batch web scraping in Hermes

Pick the way that suits you. Each takes about two minutes.

### Use the Spiral skill

1. **Send Hermes this message**: Your agent reads Spiral's skill and sets itself up.

```
set up https://spiral.ai/SKILL.md
```

2. **Sign in or paste a key**: Approve Spiral in the browser, or give the agent a key from the console.

### Use Spiral's MCP server

1. **Add the server and sign in**: Hermes opens your browser: sign in to Spiral and click Authorize. No API key. When Hermes asks Enable all tools? \[Y/n/select], answer Y: n cancels without saving the server, and select hides the tools you leave out.

```bash
hermes mcp add spiral \
  --url https://api.spiral.ai/mcp \
  --auth oauth \
  --connect-timeout 300
```

2. **Check it**: In a chat that was already open, run `/reload-mcp`.

```bash
hermes mcp list
hermes mcp test spiral
```

### Use the Spiral CLI

1. **Install and sign in**: Installs the Spiral CLI, signs you in and adds its skill to your coding agents.

```bash
npx -y spiral-cli@latest init
```

### Call the Spiral API

1. **Create an API key**: In the console, under API keys.

2. **Run Firecrawl batch web scraping**: One request; the result comes back by polling or a webhook.

```bash
curl https://api.spiral.ai/v1/runs \
  -H "Authorization: Bearer $SPIRAL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "provider": "firecrawl",
    "model": "firecrawl/batch-scrape",
    "input": {
      "urls": [
        "<urls>"
      ]
    }
  }'
```

Then paste one of the prompts above: Hermes finds Firecrawl batch web scraping on Spiral, shows the price and runs it.

Setup guide in the docs: https://spiral.ai/docs/quickstarts/hermes

## Firecrawl batch web scraping tools and prices

- [Firecrawl Batch Scrape · Many Pages](https://spiral.ai/tools/firecrawl/firecrawl/batch-scrape) (Firecrawl): $0.001746 per URL ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per URL each.

## Settings you can ask for

| Setting | Allowed values | Default | Description |
| --- | --- | --- | --- |
| proxy | basic, enhanced, auto | auto | Specifies the type of proxy to use. - **basic**: Proxies for scraping sites with none to basic anti-bot solutions. Fast and usually works. - **enhanced**: Enhanced proxies for scraping sites with advanced anti-bot solutions. Slower, but more reliable on certain sites. Billed at the same credit cost as basic. - **auto**: Firecrawl will automatically retry scraping with enhanced proxies if the basic proxy fails. Enhanced proxies carry no credit surcharge, so either way only the regular cost is billed. |
| maxAge |  | 172800000 | Returns a cached version of the page if it is younger than this age in milliseconds. If a cached version of the page is older than this value, the page will be scraped. If you do not need extremely fresh data, enabling this can speed up your scrapes by 500%. Defaults to 2 days. |
| minAge |  | — | When set, the request only checks the cache and never triggers a fresh scrape. The value is in milliseconds and specifies the minimum age the cached data must be. If matching cached data exists, it is returned instantly. If no cached data is found, a 404 with error code SCRAPE_NO_CACHED_DATA is returned. Set to 1 to accept any cached data regardless of age. |
| mobile |  | false | Set to true if you want to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots. |
| actions |  | — | Actions to perform on the page before grabbing the content |
| formats |  | [{"type":"markdown"}] | Output formats to include in the response. You can specify one or more formats, either as strings (e.g., `'markdown'`) or as objects with additional options (e.g., `{ type: 'json', schema: {...} }`). Some formats require specific options to be set. Example: `['markdown', { type: 'json', schema: {...} }]`. |

## What can you do with Firecrawl batch web scraping in Hermes?

Receive a digest of changes across selected release notes or public policy pages. Have the agent fetch the same URLs on each run and compare the returned text with its previous snapshot.

### Build a documentation reference

Collect a selected set of documentation URLs as page text, then organize the returned content into a searchable reference with links to the original pages.

### Compare public product descriptions

Read several public product pages and build a table of stated features, supported platforms, and published specifications.

### Prepare a migration review

Fetch the documentation pages relevant to a planned code change and compare their examples and requirements with the project's implementation.

### Create a release digest

Fetch selected release-note pages on a schedule and compare their text with saved snapshots to summarize newly published changes.

### Review public policy changes

Collect a fixed list of public terms or policy pages and compare successive snapshots, highlighting the passages that changed.

### Audit selected website pages

Inspect returned page text and links across an approved URL list to find inconsistent descriptions, outdated references, or unexpected navigation links.

### Compare page presentation

Collect screenshots and page text for selected pages, then review how their visible presentation matches their stated content.

## Overview

- [Firecrawl](https://spiral.ai/providers/firecrawl)
- [All Hermes guides](https://spiral.ai/integrations/hermes)

## Firecrawl batch web scraping in other agents

- [Amp](https://spiral.ai/integrations/amp/firecrawl-batch-web-scraping)
- [ChatGPT](https://spiral.ai/integrations/chatgpt/firecrawl-batch-web-scraping)
- [Claude](https://spiral.ai/integrations/claude-desktop/firecrawl-batch-web-scraping)
- [Claude Code](https://spiral.ai/integrations/claude-code/firecrawl-batch-web-scraping)
- [Cline](https://spiral.ai/integrations/cline/firecrawl-batch-web-scraping)
- [Codex](https://spiral.ai/integrations/codex/firecrawl-batch-web-scraping)
- [Cursor](https://spiral.ai/integrations/cursor/firecrawl-batch-web-scraping)
- [Devin](https://spiral.ai/integrations/devin/firecrawl-batch-web-scraping)
- [dots](https://spiral.ai/integrations/dots/firecrawl-batch-web-scraping)
- [Factory Droid](https://spiral.ai/integrations/factory-droid/firecrawl-batch-web-scraping)
- [Gemini](https://spiral.ai/integrations/gemini/firecrawl-batch-web-scraping)
- [Gemini CLI](https://spiral.ai/integrations/gemini-cli/firecrawl-batch-web-scraping)
- [Google Antigravity](https://spiral.ai/integrations/antigravity/firecrawl-batch-web-scraping)
- [goose](https://spiral.ai/integrations/goose/firecrawl-batch-web-scraping)
- [Grok](https://spiral.ai/integrations/grok/firecrawl-batch-web-scraping)
- [Grok Build](https://spiral.ai/integrations/grok-build/firecrawl-batch-web-scraping)
- [Instinct](https://spiral.ai/integrations/instinct/firecrawl-batch-web-scraping)
- [Manus](https://spiral.ai/integrations/manus/firecrawl-batch-web-scraping)
- [Muse](https://spiral.ai/integrations/muse/firecrawl-batch-web-scraping)
- [Muse Code](https://spiral.ai/integrations/muse-code/firecrawl-batch-web-scraping)
- [n8n](https://spiral.ai/integrations/n8n/firecrawl-batch-web-scraping)
- [Notion](https://spiral.ai/integrations/notion/firecrawl-batch-web-scraping)
- [OpenClaw](https://spiral.ai/integrations/openclaw/firecrawl-batch-web-scraping)
- [opencode](https://spiral.ai/integrations/opencode/firecrawl-batch-web-scraping)
- [Perplexity](https://spiral.ai/integrations/perplexity/firecrawl-batch-web-scraping)
- [Raycast](https://spiral.ai/integrations/raycast/firecrawl-batch-web-scraping)
- [VS Code](https://spiral.ai/integrations/vscode/firecrawl-batch-web-scraping)
- [Windsurf](https://spiral.ai/integrations/windsurf/firecrawl-batch-web-scraping)
- [Zapier](https://spiral.ai/integrations/zapier/firecrawl-batch-web-scraping)

## More Hermes guides

- [Exa web research](https://spiral.ai/integrations/hermes/exa-web-research)
- [Firecrawl developer search](https://spiral.ai/integrations/hermes/firecrawl-developer-search)
- [Exa web page content](https://spiral.ai/integrations/hermes/exa-web-page-content)
- [Firecrawl web page content](https://spiral.ai/integrations/hermes/firecrawl-web-page-content)
- [Exa similar web pages](https://spiral.ai/integrations/hermes/exa-similar-web-pages)
- [Firecrawl web research](https://spiral.ai/integrations/hermes/firecrawl-web-research)

## Questions about Firecrawl batch web scraping in Hermes

**Can Hermes use Firecrawl batch web scraping?**

Yes. Add Spiral to Hermes as an MCP server, and Hermes can find and run Firecrawl batch web scraping like any built-in tool.

**How much does Firecrawl batch web scraping cost in Hermes?**

$0.001746 per URL ($0.00873 with JSON output). PII redaction, a prompt injection check or lockdown: $0.006984 more per URL each. You pay from your Spiral balance as you go, with no subscription, and a failed run is not charged. Your Hermes plan is separate.

**Do I need a Firecrawl account to use Firecrawl batch web scraping in Hermes?**

No. Spiral runs Firecrawl batch web scraping on its own Firecrawl account. You sign in to Spiral once and never paste a key.

**How long does it take to set up Firecrawl batch web scraping in Hermes?**

About two minutes: add the server and sign in, then check it.

**Does Hermes wait while Firecrawl batch web scraping runs?**

Firecrawl batch web scraping runs in the background. Hermes starts the run, checks on it and gives you the json when it's ready.

**Does Firecrawl batch scraping collect an entire website?**

It processes an explicit list of URLs. Its coverage follows that list, so include each page you want represented; some websites are also unsupported.

**How fresh is the returned page content?**

Firecrawl's default cache freshness window is two days. A new request can therefore return a recent cached copy rather than newly fetched website content.

**Can one result response contain the whole batch?**

Large results require pagination: Firecrawl provides a continuation URL for the next 10 MB of data. The response also reports how many pages were successfully scraped.

**How long can I retrieve completed results?**

Batch results are available through the API for 24 hours after completion. Save the returned content within that window if you need it for later comparisons or indexing.

**Can structured JSON include hidden HTML attributes?**

JSON extraction works from the page's Markdown conversion, which preserves visible text. HTML attributes are unavailable to that extraction, so use original HTML when those values matter.

