Prints for whoever is reading

One Rust binary that prints colour and live progress for a person, one fact per line for a pipe, CI or an AI agent, and JSON for your scripts.

Install Every command

raptor crawl http://127.0.0.1:8006/

$ raptor crawl http://127.0.0.1:8006/$

Raptor 0.3.0 on 17 Oct 2026, recorded byte for byte on a real terminal and replayed in order. The replay compresses each crawl’s wait (Raptor pauses 250 ms between requests by default) and the title bar says by how much. Fernhollow is a test shop built for this site; the rest are the fixture sites in Raptor’s own repository.

Three layouts, one crawl

The same Fernhollow crawl, as a person, a pipe and a script see it. Each output stream decides for itself, so the progress can stay on screen while the summary goes to a file.

Plain, for pipes, CI and agents

One key: value fact per line: no colour, no animation, full URLs and absolute paths, the verdict first and the files last. Raptor switches to it by itself when Claude Code or another agent runs it. The terminal’s “Limping” becomes “Needs Work”: Raptor’s voice is for people, plain words for pipes.

raptor crawl http://127.0.0.1:8006/ --format plain

raptor 0.3.0 crawl http://127.0.0.1:8006/
result: ok (exit 0)
crawl: Crawl complete: 27 pages in 7.1s.
headline: 79/100: Needs Work. Here is what is holding the score back.
grade: C 79/100 (Needs Work)
why: Page error (4xx) on 1 page costs the most, about 3.7 points. Next is Dead-end canonical, about 3.7.
findings: 13 to fix, 4 notes; 94 checks passed
fix first:
1. Fix now · response_codes.internal_client_error · 1 page returns an error instead of its content
   reach: 1 page; points: +3.7
   fix: Bring each page back, or send its address to the closest live page with a 301 redirect (a permanent forwarding rule set on the server). Then update the interna…
   example: http://127.0.0.1:8006/products/storm-shell
2. Fix now · canonicals.non_indexable_target · 1 page names a preferred version that search engines cannot use
   reach: 1 page; points: +3.7
   fix: Point each canonical tag at a live page that loads normally and is not marked noindex. Usually that is the page's own address.
   example: http://127.0.0.1:8006/waterproofs/storm-shell/

The first 16 lines. The rest are the other fixes, the files and the next step.

JSON, for scripts

--json prints the summary as JSON, schema raptor.crawl-summary/1, pretty-printed in a terminal and one line in a pipe. It never lists every affected URL, so it stays small on a big crawl.

raptor crawl http://127.0.0.1:8006/ --json

{  "schema": "raptor.crawl-summary/1",  "result": "ok",  "exit_code": 0,  "raptor_version": "0.3.0",  "domain": "127.0.0.1",  "seed_url": "http://127.0.0.1:8006/",  "requested_url": "http://127.0.0.1:8006/",  "headline": "79/100: Needs Work. Here is what is holding the score back.",  "crawl": {    "pages_crawled": 27,    "html_pages": 26,    "duration_ms": 7088,    "complete": true,    "stopped_at_page_limit": false,    "urls_not_crawled": 0,    "interrupted": false,    "errored_urls": 0,    "robots_blocked_urls": 0  },  "failure": null,  "score": {    "graded": true,    "score": 79,

The first 24 of 182 lines.

How Raptor decides

The first rule that applies wins, for stdout and stderr separately.

Order	Setting	Effect
1	`--json` (crawl and inspect)	stdout prints the JSON summary
2	`--format rich` or `--format plain`	that layout
3	`--color always`, `FORCE_COLOR` set, or `CLICOLOR_FORCE` set (not `0`)	rich, in colour
4	An AI agent is detected (below)	plain
5	The stream is a TTY and `TERM` is not `dumb`	rich
6	Otherwise	plain

Every command

Each one with what it does and its real output. Run raptor <command> --help for the full flag list.

raptor crawl

Crawl a site and analyse it

network: yesinput: a URL

Crawls breadth-first from the start URL, then runs every check over the pages it collected. Every other command reads what it writes.

It writes the JSON and the self-contained HTML report beside each other, and a CSV with --csv. The tabs at the top of this page are six real runs of it.

Shell

$ raptor crawl https://example.com/

The start URL is resolved first: Raptor follows its redirect and audits where it lands. Seeding www. on a site that redirects to the bare domain crawls the site, rather than one page with every link counted as external.
Scope is the host and port: Links to other hosts are recorded, not crawled. The scheme isn’t compared, so an http:// link on an HTTPS site is still internal.
One politeness gate: Workers share one delay instead of sleeping on their own. robots.txt is obeyed (RFC 9309), and a 5xx or 429 robots.txt blocks every path, as Google treats it.
An exact page budget: --max-pages is an upper bound, not a target. Stopping there says how many URLs were left behind, and the JSON records them.
Private addresses are refused: Targets in private, loopback, link-local and cloud-metadata ranges are blocked, checking the address actually dialled. RAPTOR_ALLOW_PRIVATE_TARGETS=1 lifts it for an intranet or a local test site.

raptor inspect

Single-page audit, no crawl needed

network: yesinput: a URL

Audits one URL, no crawl needed: title, description, canonical, indexability with the directives that decided it, status, words, links, images, schema and security headers, then the findings that apply to that page, most important first.

The fastest way to confirm a finding from a crawl without running it again. A URL that redirects is reported for the page that served it, and says so.

Shell

$ raptor inspect https://example.com/pricing

raptor inspect http://127.0.0.1:8006/waterproofs/

  /// raptor inspect  http://127.0.0.1:8006/waterproofs/  static HTML  PAGE  Title        Waterproof Jackets and Trousers | Fernhollow  Description  Waterproof jackets and trousers made for sideways rain: taped               seams, proper hoods and fabric that keeps breathing on long               climbs.  Canonical    http://127.0.0.1:8006/waterproofs/  Indexable    no (index, follow, noindex)  Status       200 in 5ms  Content      134 words, readability 74.9  Links        15 internal, 0 external  Images       0 (0 with alt)  Schema       WebPage  Security     HTTPS no · HSTS no · CSP yes · X-Frame-Options yes  FINDINGS                                                   1 to fix · 1 note   1  FIX NOW  1 page both allows and blocks listing in search results               Make the robots meta tag in the page template and the               X-Robots-Tag header in the server settings agree. Remove               noindex from both on any page you want in search results.               Check: directives.conflicting_directives  NOTES  ● 1 page asks search engines not to list it · directives.noindex

raptor inspect http://127.0.0.1:8006/waterproofs/

  /// raptor inspect  http://127.0.0.1:8006/waterproofs/  static HTML  PAGE  Title        Waterproof Jackets and               Trousers | Fernhollow  Description  Waterproof jackets and               trousers made for sideways               rain: taped seams, proper               hoods and fabric that keeps               breathing on long climbs.  Canonical    http://127.0.…/waterproofs/  Indexable    no (index, follow, noindex)  Status       200 in 0ms  Content      134 words, readability 74.9  Links        15 internal, 0 external  Images       0 (0 with alt)  Schema       WebPage  Security     HTTPS no · HSTS no · CSP               yes · X-Frame-Options yes  FINDINGS               1 to fix · 1 note   1  FIX NOW  1 page both allows and               blocks listing in search               results               Make the robots meta tag in               the page template and the               X-Robots-Tag header in the               server settings agree.               Remove noindex from both on               any page you want in search               results.               Check:               directives.conflicting_               directives  NOTES  ● 1 page asks search engines not to list    it · directives.noindex

raptor link

Internal link graph: PageRank, orphans, hubs, silos

network: noinput: crawl.json

Builds the internal link graph from a finished crawl and runs PageRank over it (damping 0.85, 40 iterations): which pages your own links treat as most important, which have no links in, and how the site splits into sections.

It reads the crawl file and fetches nothing, so you can re-run it with a different --top for free. On this fixture, the privacy policy outranks the best-selling category; the home page charts it.

Shell

$ raptor link crawl.json$ raptor link crawl.json --json --top 50

raptor link crawl-127.0.0.1-2026-10-17.json --top 30

  /// raptor link  crawl-127.0.0.1-2026-10-17.json  27 nodes · 341 edges · 1 component · avg out-degree 12.6  TOP PAGES BY INTERNAL AUTHORITY (PAGERANK)    1  0.0680  http://127.0.0.1:8006/tents (25 in / 16 out)    2  0.0676  http://127.0.0.1:8006/backpacks (25 in / 14 out)    3  0.0676  http://127.0.0.1:8006/sleeping-bags (25 in / 14 out)    4  0.0676  http://127.0.0.1:8006/journal (25 in / 14 out)    5  0.0673  http://127.0.0.1:8006/ (25 in / 13 out)    6  0.0669  http://127.0.0.1:8006/about (25 in / 12 out)    7  0.0669  http://127.0.0.1:8006/delivery (25 in / 12 out)    8  0.0669  http://127.0.0.1:8006/sale (25 in / 12 out)    9  0.0669  http://127.0.0.1:8006/returns (25 in / 12 out)   10  0.0669  http://127.0.0.1:8006/privacy (25 in / 12 out)   11  0.0669  http://127.0.0.1:8006/terms (25 in / 12 out)   12  0.0669  http://127.0.0.1:8006/contact (25 in / 12 out)   13  0.0669  http://127.0.0.1:8006/en-us (25 in / 12 out)   14  0.0109  http://127.0.0.1:8006/waterproofs (3 in / 15 out)

The first 20 of 54 lines. The best-selling category is 14th of 27, behind every footer page.

raptor sitemap

XML sitemap of the indexable pages

network: noinput: crawl.json

Writes an XML sitemap of the crawl’s indexable pages: 2xx HTML that isn’t noindex. A 404, a redirect, a PDF or a noindex page is left out.

If nothing qualifies, it writes nothing and exits non-zero, rather than an empty <urlset> that Search Console would read as an empty sitemap. Past the 50,000-URL limit it writes a sitemap index and chunk files, because Google rejects an oversized sitemap whole.

Shell

$ raptor sitemap crawl.json -o sitemap.xml

sitemap.xml from the Fernhollow crawl (the first 10 of 69 lines)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>http://127.0.0.1:8006/</loc>
  </url>
  <url>
    <loc>http://127.0.0.1:8006/about/</loc>
  </url>
  <url>
    <loc>http://127.0.0.1:8006/backpacks/</loc>

raptor serve

REST API and/or MCP server

network: yesinput: —

The REST API on 127.0.0.1:3001, or the MCP server over stdio with --mcp. Every Raptor process on the same RAPTOR_HOME shares one job store, so both can run at once.

Auth is off unless you set --api-key or RAPTOR_API_KEY, which is why it binds to 127.0.0.1, and why it warns you when it starts. The MCP page covers the tools.

Shell

$ raptor serve$ raptor serve --mcp

raptor serve

2026-10-17T14:45:24.508024Z  WARN raptor_server::auth: No RAPTOR_API_KEY set — API is unauthenticated. Set RAPTOR_API_KEY for production use.  Raptor Server v0.3.0  Listening on http://127.0.0.1:30012026-10-17T14:45:24.508776Z  INFO raptor_server: raptor-server listening on 127.0.0.1:3001

raptor version

The version and the features this build has

network: noinput: —

The version, and the optional features this build was compiled with.

Shell

$ raptor version

raptor version

  /// raptor 0.3.0  built with: server

raptor

On its own, in a terminal, the home screen: the four commands to start with and your last five crawls, each with its grade, score, page count, age and report path. With no crawls yet, as here, it suggests one. In a pipe or for an agent it prints the help instead.

Shell

$ raptor

raptor

  /// raptor 0.3.0  Crawl a site, grade it, and see what to fix first.  START  raptor crawl <url>          Crawl a site and grade it  raptor inspect <url>        Check a single page  raptor link <crawl.json>    Map internal links (PageRank, orphans, hubs)  raptor serve --mcp          Use Raptor from Claude and other AI agents  RECENT CRAWLS  No crawls yet. Try: raptor crawl https://example.com  raptor --help lists every command and option.

raptor

  /// raptor 0.3.0  Crawl a site, grade it, and see what to fix first.  START  raptor crawl <url>  raptor inspect <url>  raptor link <crawl.json>  raptor serve --mcp  RECENT CRAWLS  No crawls yet. Try: raptor crawl https://example.com  raptor --help lists every command and option.

With the Cloudflare feature

Three commands need a build with --features cloudflare and a Cloudflare account. The default build leaves them out, and running one says so and gives the cargo command that adds them. They aren’t shown running here, because this site’s captures come from the default build.

raptor screenshot: Full-page or element screenshots through Cloudflare Browser Rendering: --full-page, --selector, --viewport WxH, --format png|jpeg.
raptor extract: AI structured extraction from a page through Cloudflare, with an SEO preset or your own --prompt or --schema.
raptor setup cloudflare: Asks for the account ID and token, checks them with Cloudflare, and writes them to $RAPTOR_HOME/.env with 0600 permissions, never to the working directory.

Flags worth knowing

From the crawl's own settings line: 3 workers, 250 ms apart, up to 500 pages, depth 10, robots.txt obeyed. These change that.

Crawling

Flag	Effect
`--max-pages N`	An exact upper bound. Stopping there still exits 0, and says how many URLs it left behind.
`--depth N`	How far from the start URL to follow links.
`--concurrency N`, `--delay MS`	Workers share one delay: three workers at 250 ms make one request every 250 ms.
`--follow-subdomains`	Widens scope from the start URL's host to subdomains of the same registrable domain.
`--ignore-robots`	Crawls what robots.txt disallows. Labelled not recommended: it's for sites you control.
`--backend playwright`	Renders pages in local Chrome, for sites that build content with JavaScript. Needs a build with the playwright feature.
`--pagespeed`	Adds Core Web Vitals from Google's PageSpeed API (`--pagespeed-key` or `PAGESPEED_API_KEY`; `--pagespeed-sample N` limits it to N pages).

Output files

Flag	Effect
`--no-html`	JSON only — what you want when an agent or a script is the consumer
`--csv`	Also write a CSV, one row per page
`--summary-only`	Metadata, issues and summary; drops the per-page array
`--include-body-text`	Include page text in the JSON (excluded by default; it dominates file size)
`--json-compact`	Minified JSON
`--json`	Print the JSON summary on stdout instead of text (the crawl JSON file is still written)

Your agency’s brand on the report

Flag or variable	Effect
`--brand <FILE>`	Put your agency's name, logo and colour on the HTML report, from a `brand.toml`. See report.md
`RAPTOR_BRAND`	The path of a `brand.toml` to use when `--brand` is not given
`--no-brand`	Use the plain Raptor brand, ignoring `--brand`, `RAPTOR_BRAND` and `brand.toml` in Raptor's home folder

In CI

Findings never fail a run on their own. Gates do: a failed gate still writes the files, and exits 4. A crawl with nothing to grade exits 3 before any gate is checked, so a blocked or unreachable site never passes by accident.

Flag	Fails the run when
`--min-score N`	the score is below `N` (0 to 100)
`--fail-on PRIORITY`	any finding is at that priority or more urgent: `fix-now`, `high`, `medium` or `low`

Shell

$ raptor crawl https://staging.example.com --json --min-score 80 --fail-on high > summary.json

raptor crawl http://127.0.0.1:8006/ --json --min-score 80 --fail-on fix-now (excerpt)

"result": "quality_gate_failed",
"exit_code": 4,
"gates": [
  {"gate": "min_score", "status": "failed", "detail": "score 79 is below --min-score 80"},
  {"gate": "fail_on", "status": "failed", "detail": "3 findings at Fix now priority or more urgent (--fail-on fix-now)"}
]

Exit codes

The same table for every command.

Code	Meaning
`0`	it worked (findings never fail a run on their own; stopping at `--max-pages` is still 0)
`1`	runtime error: an unwritable output path, a request that failed, a backend this build lacks, a brand file Raptor cannot use
`2`	usage error: an unknown flag or value, or a URL without `https://`
`3`	nothing to grade: the start URL could not be reached, robots.txt disallows it, every page answered with an HTTP error, or the crawl ran but nothing could be graded (a bot wall, or an app shell with no readable content)
`4`	a quality gate failed (`--min-score`, `--fail-on`)
`130`	interrupted with Ctrl-C (what was crawled is saved)

Ctrl-C keeps what it found

Press it once and Raptor lets the requests in flight finish, analyses the pages it has, writes the files and exits 130. The summary says the crawl was interrupted and gives the command that runs it again. Press it twice to quit at once without saving.

raptor crawl http://127.0.0.1:8006/ (Ctrl-C after 2.6 s)

  /// raptor crawl  http://127.0.0.1:8006/  3 workers · 250ms apart · up to 500 pages · depth 10 · obeys robots.txt  ● robots.txt: 1 rule  ▲ Stopping. Finishing the pages in flight, then saving what was crawled. Press Ctrl-C again to quit without saving.  ✓ Fetching pages: 9 pages                                              2.5s  ✓ Analysing: 9 pages                                                    3ms  ✓ Saving: JSON and HTML report                                         13ms  ▲ Hunt interrupted. 9 pages saved.   B   87/100 · Healthy pack  ────────────────────────────────────────────────────────────────────────────  Not extinct, not apex. Here's what's holding you back.  Index-rule conflict on 1 page costs the most, about 11.1 points. Next is  Redirect chain, about 1.3.  FIX FIRST                                                 2 to fix · 2 notes   1  FIX NOW  1 page both allows and blocks listing in search results               1 page · +11.1               Make the robots meta tag in the page template and the               X-Robots-Tag header in the server settings agree. Remove               noindex from both on any page you want in search results.               e.g. /waterproofs/               Check: directives.conflicting_directives   2  MEDIUM   1 address redirects more than once               1 URL · +1.3               Point each redirect straight at the final address in your               server, CDN or CMS settings. Then update links, and the sitemap               (the file that lists your pages for search engines), to use the               final address.               e.g. /shop/sale.html               Check: response_codes.internal_redirect_chain  + 2 more findings in the report.  LIMITS   The crawl was stopped before it finished. Findings cover the 9           pages fetched before it stopped.  REPORT   crawl-127.0.0.1-2026-10-17.html  DATA     crawl-127.0.0.1-2026-10-17.json  NEXT     Run the crawl again to finish it: raptor crawl           'http://127.0.0.1:8006/'

While pages are still being fetched, the crawl stops there and keeps what it has.
Once every page is fetched, it skips the sitemap check and says so.
During --pagespeed, it skips the PageSpeed tests and saves the crawl as it is.
While the crawl is analysed or saved, nothing is left to stop: it’s saved whole.

Environment variables

Settings that live outside a single command.

Variable	Purpose
`RAPTOR_HOME`	Data directory (default `~/.raptor`): job database, job output, credentials
`RAPTOR_API_KEY`	Bearer token required by the REST API and MCP HTTP routes
`RAPTOR_ALLOW_PRIVATE_TARGETS`	Allow crawling private/loopback addresses (intranet audits, local fixtures)
`RAPTOR_CORS_ORIGINS`	Comma-separated CORS allowlist; CORS is closed by default
`RAPTOR_MAX_CONCURRENT_CRAWLS`	Concurrent crawl cap for the server (default 3)
`CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN`	Cloudflare rendering credentials
`PAGESPEED_API_KEY`	Google PageSpeed API key
`RAPTOR_BRAND`	The `brand.toml` the HTML report uses when `--brand` is not given
`RUST_LOG`	Log filter, which wins over `-q` and `-v` (for example `RUST_LOG=raptor_core::crawler=debug`). Without it a crawl logs warnings, `-v` adds Raptor's own debug lines and `-vv` every library's. In MCP mode logs go to stderr, never stdout
`NO_COLOR`, `FORCE_COLOR`, `CLICOLOR`, `CLICOLOR_FORCE`	Colour standards; see Output
`CLAUDE_CODE_CHILD_SESSION`, `CLAUDECODE`, `AI_AGENT`, `AGENT`, `CURSOR_AGENT`	Tell Raptor an AI agent is reading, which gets plain output; see Output
`COLORTERM`	`truecolor` or `24bit` paints the `///` mark in Raptor's exact green
`COLORFGBG`	A light background (`…;15` or `…;7`) switches the mark to the darker green

The full commands reference