Cloudflare Browser Rendering Integration

Changed in 0.3.0: the whole-site raptor crawl --backend cloudflare backend was removed. Cloudflare Browser Rendering now powers raptor screenshot, raptor extract, and raptor inspect --rendered / --compare. For JS-rendering during a full crawl, build with the playwright feature and use --backend playwright. Sections below that describe --backend cloudflare are retained for historical context and no longer reflect the CLI. (See ROADMAP.md decision item D-5.)

Overview

Modern websites built with React, Angular, Vue, Next.js, and other JavaScript frameworks often return skeleton HTML to traditional HTTP crawlers. The actual content only appears after JavaScript executes in a browser. This means a standard crawl sees empty <div id="root"></div> tags instead of real page content, and Raptor's 114+ SEO analyzers have nothing meaningful to work with.

Cloudflare Browser Rendering solves this by running a full Chromium instance in the cloud. When Raptor uses the Cloudflare backend, every page is rendered with JavaScript before analysis begins. The result is the same fully-hydrated DOM that a real user (and Googlebot) sees.

This integration is completely optional. Raptor works fully without it. If your sites serve server-rendered HTML, the default local crawl mode is faster and has no external dependencies. Cloudflare Browser Rendering is for when you need to audit JS-heavy SPAs that are invisible to traditional crawlers.

The integration unlocks four capabilities:

• JS-rendered crawling -- crawl SPAs and get real content for all 114+ analyzers
• Screenshots -- full-page or element-level captures at any viewport size
• Single-page SEO audit -- static, JS-rendered, or side-by-side comparison audit of any URL
• Data extraction -- pull structured business data from pages using a built-in SEO schema or custom prompts

Getting Started

Requirements

• A free Cloudflare account (no paid plan required)
• Your Cloudflare Account ID and an API token with Browser Rendering permissions

Step-by-step setup

1. Sign up at cloudflare.com if you don't have an account.

2. Create an API token with the "Browser Rendering - Edit" permission:

   • Go to dash.cloudflare.com/profile/api-tokens
   • Click "Create Token"
   • Under "Custom token", click "Get started"
   • Add the permission: Account > Browser Rendering > Edit
   • Complete the token creation and copy the token value

3. Find your Account ID:

   • Log into the Cloudflare dashboard
   • Your Account ID is visible in the URL of any dashboard page: dash.cloudflare.com/<ACCOUNT_ID>/...
   • It's also shown on the right sidebar of your account's main page

4. Configure Raptor using either the setup command or environment variables:

   Option A: Interactive setup (recommended)

   ShellCopy

   raptor setup cloudflare

   This walks you through entering your Account ID and API Token, validates them against the Cloudflare API, and stores the credentials locally.

   Option B: Environment variables

   ShellCopy

   export CLOUDFLARE_ACCOUNT_ID="your-account-id"
   export CLOUDFLARE_API_TOKEN="your-api-token"

   Option C: CLI flag overrides

   Every Cloudflare command accepts --cf-account-id and --cf-api-token flags that override environment variables and stored credentials:

   ShellCopy

   raptor inspect https://example.com --rendered --cf-account-id abc123 --cf-api-token your-token

Environment variables

Verifying the setup

After configuring, the Raptor dashboard (raptor with no arguments) shows cloudflare with a checkmark in the Toolkit section:

  Toolkit:
    crawl  entity  link  cloudflare

Free Tier & Pricing

Cloudflare Browser Rendering includes a free tier:

• 10 minutes of browser rendering time per day
• Resets at midnight UTC

What that means in practice

Browser time depends on page complexity. Simple marketing pages render in 1-2 seconds. Heavy SPAs with lots of API calls and animations can take 5-10 seconds. As a rough guide:

Tracking usage

Raptor displays browser time consumed after every Cloudflare crawl:

  Browser time: 42.3s used

All render: true requests are billed under Cloudflare's Browser Rendering pricing. The free tier is generous enough for most small-to-medium site audits.

Commands Reference

raptor crawl --backend cloudflare

Crawl a website using Cloudflare Browser Rendering. Every page is rendered in Chromium before Raptor's analyzers run, so JS-rendered content is fully visible.

raptor crawl https://my-react-app.com --backend cloudflare --max-pages 100

How it works:

1. Raptor submits a crawl job to the Cloudflare /crawl API
2. Cloudflare discovers and renders pages in Chromium, following internal links
3. Raptor polls the job status, showing progress as pages complete
4. Once finished, Raptor retrieves all rendered HTML via cursor pagination
5. Raptor runs the same 114+ analyzers on the rendered HTML
6. Unless --skip-header-check is set, Raptor makes parallel GET requests to fill in security headers (HSTS, CSP, X-Frame-Options, etc.) since Cloudflare doesn't return HTTP response headers

Full flags:

Flags that do not apply in Cloudflare mode:

These flags are accepted but have no effect when --backend cloudflare is set, because Cloudflare controls the crawl execution:

Example with all relevant options:

raptor crawl https://my-spa.com \
  --backend cloudflare \
  --max-pages 200 \
  --depth 5 \
  --follow-subdomains \
  --include-body-text \
  --csv

raptor screenshot <url>

Capture a screenshot of any web page via Cloudflare Browser Rendering. The page is fully rendered in Chromium before the screenshot is taken.

raptor screenshot https://example.com --full-page

Flags:

Examples:

# Full-page screenshot at default 1920x1080 viewport
raptor screenshot https://example.com --full-page

# Mobile viewport, JPEG format with quality control
raptor screenshot https://example.com --viewport 375x812 --format jpeg --quality 80

# Capture a specific element
raptor screenshot https://example.com --selector "#hero-section" --output hero.png

# Save to a specific path
raptor screenshot https://example.com --full-page --output screenshots/homepage.png

raptor inspect <url>

Single-page SEO audit with three modes. Static mode works without Cloudflare credentials. Rendered and compare modes use Cloudflare Browser Rendering.

Modes:

raptor inspect https://example.com                    # static single-page audit
raptor inspect https://example.com --rendered         # CF JS-rendered audit
raptor inspect https://example.com --compare          # side-by-side static vs rendered

Compare mode output example:

  Raptor v0.2.0 — inspect (compare)
  https://spa-site.com

  ● Fetching static HTML...                                [00:01]
  ● Rendering via Cloudflare...                            [00:04]
  ● Running analyzers...                                   [00:01]
  ● Comparing results...                                   [00:00]

  ✓ Inspect complete — 4 differences found

  ┌────────────────────┬──────────────┬──────────────┐
  │ Check              │ Static       │ Rendered     │
  ├────────────────────┼──────────────┼──────────────┤
  │ Title              │ (empty)      │ My SPA App   │
  │ Meta description   │ (empty)      │ Welcome to…  │
  │ H1 count           │ 0            │ 1            │
  │ Word count          │ 12           │ 847          │
  └────────────────────┴──────────────┴──────────────┘

  Output:
    HTML  inspect-spa-site.com-2026-04-03.html

Flags:

Examples:

# Static audit (no Cloudflare needed)
raptor inspect https://example.com

# Rendered audit, wait for dynamic content
raptor inspect https://spa-site.com --rendered --wait-for "#app-loaded"

# Compare mode with custom output path
raptor inspect https://spa-site.com --compare --output audit-comparison.html

# Override Cloudflare credentials inline
raptor inspect https://spa-site.com --rendered \
  --cf-account-id abc123 --cf-api-token your-token

raptor extract <url>

Extract structured data from a web page using Cloudflare's Browser Rendering /json endpoint. The page is rendered in Chromium, then data is extracted based on a schema or prompt.

Default behavior (SEO preset):

With no --prompt or --schema flag, Raptor uses a built-in SEO extraction schema that pulls:

• Business name and type
• Phone number and email
• Address (street, city, state, ZIP)
• Services offered
• Service areas
• Social media profiles
• Business hours

raptor extract https://joesplumbing.com

Example output:

  Business Name:  Joe's Plumbing & Drain
  Business Type:  Plumber
  Phone:          (555) 123-4567
  Email:          info@joesplumbing.com
  Address:        123 Main St, Springfield, IL, 62701
  Services:       Drain cleaning, Water heater repair, Pipe installation, Emergency plumbing
  Service Areas:  Springfield, Decatur, Champaign, Bloomington
  Hours:          Mon-Fri 7am-6pm, Sat 8am-2pm
  Social:
    facebook: https://facebook.com/joesplumbing
    yelp: https://yelp.com/biz/joes-plumbing-springfield

Flags:

Examples:

# Default SEO preset extraction
raptor extract https://joesplumbing.com

# Custom prompt
raptor extract https://example.com --prompt "Extract all product names and prices"

# Custom schema from file
raptor extract https://store.com --schema product-schema.json --output products.json

# Wait for dynamic content to load before extracting
raptor extract https://spa-app.com --wait-for ".product-list"

Using a custom schema:

Create a JSON file following the Cloudflare Browser Rendering response format schema:

{
  "type": "json_schema",
  "schema": {
    "type": "object",
    "properties": {
      "product_name": { "type": "string" },
      "price": { "type": "string" },
      "description": { "type": "string" },
      "availability": { "type": "string" }
    }
  }
}

Then pass it with --schema:

raptor extract https://store.com --schema product-schema.json

How It Works

Local crawl (default)

Raptor  --(HTTP GET)-->  Website
        <--(HTML)------
        --> Parse HTML
        --> Run 114+ analyzers
        --> Generate report

Raptor fetches raw HTML directly using reqwest. Fast, lightweight, no external dependencies. Works perfectly for server-rendered sites. No JavaScript is executed.

Cloudflare crawl

Raptor  --(submit job)-->  Cloudflare Browser Rendering API
        <--(job ID)------
        --(poll status)-->  CF renders pages in Chromium (JS executes)
        <--(progress)----
        --(retrieve)---->  CF returns rendered HTML
        <--(HTML)--------
        --> Run same 114+ analyzers
        --> Generate report

1. Job submission: Raptor sends the seed URL, max pages, depth, and options to the Cloudflare /crawl endpoint. Cloudflare returns a job ID.

2. Polling: Raptor polls every 5 seconds until the job reaches a terminal state (completed, errored, cancelled). During polling, Raptor emits progress updates showing pages crawled vs. total discovered.

3. Retrieval: After completion, Raptor paginates through all results using cursor-based pagination, deduplicating by URL.

4. Security headers: Cloudflare returns rendered HTML but not HTTP response headers. Raptor makes parallel GET requests to each page's URL to fill in security headers (HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy). This step can be skipped with --skip-header-check.

5. Analysis: The rendered HTML is passed through the same parser and 114+ analyzers used in local mode. Reports are identical in format.

JS rendering detection

After a local crawl, Raptor checks the results for signs of JS-rendered content. If more than 25% of pages returned HTTP 200 but have zero word count and empty body text, Raptor flags this in the CLI output and the report, and points to --backend playwright, which renders pages in local Chrome. That backend needs a build with the playwright feature (cargo build --release --features playwright); the release binaries do not include it, and a build without it says so in the hint.

Known Limitations

• No redirect chain data. Cloudflare returns the final URL after all redirects, but does not expose the redirect chain. Redirect-related issues (e.g., long redirect chains) cannot be detected in CF mode.

• Response times are approximate. In CF mode, per-page response times come from the parallel header fetch, not from the actual page render. They reflect server response time, not the full JS rendering time.

• --ignore-robots is not supported. Cloudflare always respects robots.txt. Pages disallowed by robots.txt will be skipped and reported as "disallowed" in the crawl results.

• Fixed user-agent. Cloudflare uses CloudflareBrowserRenderingCrawler/1.0 as the user-agent for rendered requests. The --user-agent flag does not apply in CF mode (it is used for the parallel header requests only).

• Screenshots and extract are standalone commands. They are not embedded into crawl reports. You run them separately against individual URLs.

• 10-minute daily limit on the free tier. Large sites may require multiple days or a paid plan. Raptor reports browser time used so you can plan accordingly.

Troubleshooting

"Cloudflare auth failed"

Your API token is missing, expired, or lacks the correct permission.

Fix:

• Verify your token at dash.cloudflare.com/profile/api-tokens
• Ensure it has the "Browser Rendering - Edit" permission
• Run raptor setup cloudflare to re-enter and validate credentials
• If using environment variables, check they are exported in your current shell

"Daily browser time limit reached"

You've used all 10 minutes of free-tier browser rendering time for the day.

Fix:

• The limit resets at midnight UTC
• Reduce --max-pages to stay within the budget on future crawls
• Use the local backend (raptor crawl <url> without --backend cloudflare) as a fallback for server-rendered pages
• Consider a paid Cloudflare plan for higher limits

"Crawl timed out"

The crawl job did not complete within the 10-minute deadline.

Fix:

• Reduce --max-pages (try 50-100 for large sites)
• Reduce --depth to limit how far from the seed URL the crawl goes
• Check that the target site is reachable and not blocking Cloudflare's IP ranges

"Cloudflare API unreachable"

Raptor cannot connect to the Cloudflare API.

Fix:

• Check your internet connection
• Verify that api.cloudflare.com is not blocked by a firewall or proxy
• Use the local backend as a fallback: raptor crawl <url>

"Rate limited by Cloudflare"

This is typically transient. Raptor automatically retries with exponential backoff when rate-limited.

Fix:

• If it persists, wait a few minutes and try again
• Reduce --max-pages to generate less API traffic

Cloudflare credentials not detected

If raptor shows cloudflare without a checkmark in the Toolkit section, credentials are not configured.

Fix:

• Run raptor setup cloudflare to configure interactively
• Or set CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN environment variables
• Or pass --cf-account-id and --cf-api-token flags directly to commands