CLI commands

Every command is raptor <command>. Run raptor <command> --help for the exact flag list; this document explains what each command does and how it works, which the help text does not.

Run raptor on its own in a terminal for the home screen: the commands to start with and your recent crawls.

Output: rich, plain and JSON

Raptor prints for whoever is reading. In an interactive terminal window (a "TTY") you get the rich layout: colour, a live progress region while the crawl runs, and a summary that leads with the grade and what to fix first. Through a pipe (raptor crawl … | tee log.txt), in CI, or when an AI agent runs it, you get the plain layout: one key: value fact per line, no colour, no progress animation, full URLs and absolute file paths, with the verdict first and the files last. --json prints a small JSON summary instead, for scripts and agents that parse output.

Progress and logs always go to stderr, and results to stdout, so raptor crawl … > summary.txt captures the summary while the progress stays on screen.

Each output stream decides on its own, in this order (the first rule that applies wins):

Colour in the rich layout follows the usual standards: --color always|never first, then NO_COLOR (set to anything turns colour off), FORCE_COLOR, and CLICOLOR=0. NO_COLOR on its own keeps the rich layout, just without colour. Plain and JSON output never carry colour or terminal control codes.

An AI agent is detected when CLAUDE_CODE_CHILD_SESSION=1, when CLAUDECODE=1 and stdout is not a TTY (an IDE terminal sets CLAUDECODE=1 on a real terminal, where a person reads), or when AI_AGENT, AGENT or CURSOR_AGENT is set. An agent gets plain output even on a terminal.

--format goes after crawl or inspect (raptor crawl https://example.com --format plain), or before any subcommand (raptor --format plain link crawl.json). raptor screenshot --format is still the image format, png or jpeg.

-q (--quiet) prints only errors: no progress, no summary and no log warnings, though a crawl that has nothing to grade still names its cause on stderr; the files are still written. -v adds a line per page and Raptor's own debug log; -vv logs every library. --color, -q and -v work before or after any subcommand.

Exit codes

An exit code is the number a command hands back to the shell or CI when it ends; 0 means success. Every Raptor command uses the same table:

raptor crawl

raptor crawl https://example.com/ -o crawl.json

Crawls a site breadth-first from the seed URL, then runs every analyser over the collected pages. This is the command the others consume.

How it works

The seed is resolved first. Before anything else, Raptor follows the seed's own redirect and treats where it lands as the site under audit. Seeding https://www.example.com/ on a site that canonicalises to the apex used to classify every discovered link as external and end the crawl after one page — with a passing grade and no warning. The URL that finally served a page is that page's identity throughout; the URL as requested is preserved in its redirect chain.

Scope is the seed's host and port. A link to a different host is external and is recorded but not crawled. --follow-subdomains widens this to subdomains of the same registrable domain. Scheme is deliberately not compared, so an http:// link on an HTTPS site is still internal.

Politeness is a single shared gate, not a per-worker sleep: --concurrency workers share one --delay interval, so three workers with a 250 ms delay produce one request per 250 ms, not three. robots.txt is fetched and honoured (RFC 9309), and a Crawl-delay larger than --delay wins. A robots.txt that answers 5xx or 429 makes every path disallowed, as Google does (RFC 9309 says so for 5xx only). A robots.txt that cannot be fetched at all (DNS, refused, TLS, timeout) does not block the crawl: Raptor crawls without rules and lets the start page decide whether the site is reachable. That is a deliberate departure from RFC 9309 §2.3.1.4, and Google treats an unreachable robots.txt as blocking the whole site, so the robots line says so. --ignore-robots exists and is labelled not recommended for good reason: it is for auditing sites you control.

Budget. --max-pages is an exact upper bound, not approximate. --depth bounds distance from the seed. A crawl that reaches --max-pages with URLs still waiting prints "Stopped at the page limit" with how many it left behind, and the JSON records them as stopped_at_page_limit and urls_not_crawled.

Safety. Crawl targets are checked against private, loopback, link-local and cloud-metadata ranges to prevent SSRF, with the DNS resolution that is vetted being the one actually dialled. Set RAPTOR_ALLOW_PRIVATE_TARGETS=1 to audit an intranet or a local fixture.

What it prints

In a terminal the crawl opens with a header (the /// mark, the URL and the settings on one quiet line) and a live progress region: the phase, pages done and queued, a tally of answers by status, the rate and the page in flight. Each finished phase collapses into one ✓ line with its time, and robots.txt and fetch errors print as they happen (the first ten errors, then one line saying the summary counts the rest).

The end of the run leads with the grade and the score, one sentence on where the points went, and Fix first: the five most important things to fix, in plain words, each with how far it reaches, the points it is costing, how to fix it, an example page and the stable check id (such as links.broken_internal). Below that come the run's limits, any quality gates, the files written and what to do next.

Piped, the same facts come one per line:

raptor 0.3.0 crawl https://example.com/
result: ok (exit 0)
crawl: Crawl complete: 125 pages in 31s.
headline: 83/100: Good. Here is what is holding the score back.
grade: B 83/100 (Good)
why: Render-blocking script on every page costs the most, about 5.7 points.
findings: 21 to fix, 2 notes; 96 checks passed
fix first:
1. High · javascript.js_render_blocking · 125 pages wait for a script before showing anything
   reach: Every page — fix once; points: +5.7
   fix: Add defer to the script tag (or async if no other script depends on it), …
   example: https://example.com/ (https://example.com/assets/app.js)
…
next: Map internal links: raptor link /home/me/crawl-example.com-2026-10-17.json
files:
json: /home/me/crawl-example.com-2026-10-17.json
html: /home/me/crawl-example.com-2026-10-17.html

--json prints the same summary as one line of JSON (pretty-printed in a terminal), with the schema name raptor.crawl-summary/1: the result and exit code, the headline, the score, the counts, the Fix first items with their copy, the limits, the gates, the next steps and the absolute paths of the files. It never lists every affected URL, so it stays small on a large crawl; the full detail is in the crawl JSON file.

Output files

By default writes the JSON and a self-contained HTML report beside it, sharing the stem of --output: -o crawl.json produces crawl.json, crawl.html, and crawl.csv with --csv.

The HTML report is genuinely self-contained: no CDN, no external fonts, nothing fetched when it is opened. It is safe to email to a client. The report embeds every crawled page; only per-URL detail strings are capped (100 per finding), and the report says so.

Branding

A brand file can also set the voice: with voice = "plain" the terminal's rich output speaks plainly too.

Quality gates

Gates let CI fail a build when the site gets worse. A run with a failed gate still writes its files and exits 4.

Notes never fail a run (--fail-on note is refused). A crawl with nothing to grade exits 3 before any gate is checked, so a blocked or unreachable site never passes by accident.

raptor crawl https://staging.example.com --json --min-score 80 --fail-on high > summary.json

Stopping a crawl

Press Ctrl-C once to stop: Raptor lets the requests already in flight finish, analyses the pages it has, writes the files and exits 130. The summary says the crawl was interrupted and gives the command that runs it again. What one press can stop depends on when it lands:

• while pages are still being fetched, the crawl stops there and keeps what it has;
• once every page is fetched, it skips the sitemap check (the summary says so, exit 130);
• during --pagespeed, it skips the PageSpeed tests and saves the crawl as it is (exit 130);
• while the crawl is analysed or saved, nothing is left to stop: the crawl is saved whole and the run ends with its usual result.

Press Ctrl-C a second time to quit at once without saving.

When a crawl cannot see the site

When Raptor gets nothing it can grade, the summary names the cause instead of a grade:

• the start URL could not be reached, and why: the name did not resolve (DNS), the secure connection failed (TLS), the connection was refused, it timed out, the connection closed before a response, or the address is private and blocked;
• robots.txt disallows the start URL, or robots.txt answered 5xx or 429, so every path is disallowed;
• every page Raptor fetched answered with an HTTP error, counted by status, as in HTTP 403 on 1 of 1.

result: nothing to grade (exit 3)
failure: Could not reach https://example.com/: connection refused (…). Nothing was crawled.

Only the JSON is written, because it records what failed. crawl_metadata.seed_error holds the start URL's failure (url, kind such as dns, tls, timeout or refused, and detail), and crawl_metadata.robots holds what robots.txt said. There is no HTML report and no CSV. If an earlier run left them at the paths this run would have used, they are removed, and a cleanup: Removed … line names each one, so an old report never sits beside a crawl that reached nothing. The run exits 3. With -q the cause still prints, on stderr.

Rendering and Core Web Vitals

--backend playwright renders pages in local Chrome instead of fetching raw HTML — needed for sites that inject their content, title or canonical client-side. Requires the playwright feature. The release binaries are built without it, so build Raptor from source with cargo build --release --features playwright to get it. --pagespeed adds Core Web Vitals via the Google PageSpeed API (--pagespeed-key, or PAGESPEED_API_KEY); --pagespeed-sample N limits it to N pages, since the API is rate-limited and one call per page on a large crawl is slow.

raptor link

raptor link crawl.json            # terminal summary
raptor link crawl.json --json     # full report as JSON

Builds the internal link graph from a completed crawl and reports PageRank, orphan pages, hubs, silos and connected components. It performs no network I/O.

See link-graph.md for what each number means and how it is computed — that document is the one to read before acting on the output.

--top N bounds every ranked list (default 15). The report always carries the true totals (orphan_page_count, silo_count) alongside the truncated lists.

raptor sitemap

raptor sitemap crawl.json -o sitemap.xml

Writes an XML sitemap containing the crawl's indexable pages: 2xx HTML that is not noindex. A 404, a redirect, a PDF or a noindex page is excluded.

If nothing qualifies — or the crawl was run with --summary-only, so there is no page array to read — it writes nothing and exits non-zero, saying which of the two applies. It will not produce a schema-valid-looking empty <urlset>, which Search Console reads as an empty sitemap.

Above the 50,000-URL protocol limit it emits a <sitemapindex> plus chunk files rather than one oversized file, which Google rejects whole.

raptor inspect

raptor inspect https://example.com/pricing

Audits one URL and prints a single-page report: title, meta description, canonical, indexability with the directives that decided it, status, word count, readability, links, images, schema types, security headers, and the findings that apply to that page, most important first and in the same plain words as a crawl, with Notes last.

A page that answers with an error (a 404, say) is still inspected, and the report opens by saying it is gone. A page that never answers exits 3, with the same message a crawl gives for an unreachable start URL. --json prints a JSON summary (schema raptor.inspect-summary/1) instead of text, and -o page.json still saves the full result, in its old shape.

Useful for checking one page quickly, and for confirming a finding from a crawl without re-running it. Inspecting a URL that redirects reports on the page that served it and still tells you about the redirect.

With the cloudflare feature: --rendered audits the JS-rendered DOM, and --compare shows static vs rendered side by side — the fastest way to see what a crawler misses on a JS-heavy site.

raptor serve

raptor serve                 # REST API on 127.0.0.1:3001
raptor serve --mcp           # MCP server over stdio

See mcp.md for the tools, the job lifecycle and the REST endpoints.

Two things worth knowing here:

• Auth is off unless you set a key. --api-key, or RAPTOR_API_KEY in the environment. With no key, every route except /api/v1/health is open to anything that can reach the port — which is why the default bind is 127.0.0.1, not 0.0.0.0.
• Jobs are shared state. Every Raptor process on the same RAPTOR_HOME (default ~/.raptor) reads the same SQLite job store. Running an MCP server and a REST server at once is supported; jobs are owned and heartbeated, so one process no longer reclaims another's running work.

raptor screenshot

Full-page or element screenshots through Cloudflare Browser Rendering. Requires the cloudflare feature and credentials. --full-page, --selector, --viewport WxH, --format png|jpeg.

raptor extract

AI structured extraction from a page through Cloudflare, with an SEO preset by default or your own --prompt / --schema. Requires the cloudflare feature and credentials.

raptor setup

raptor setup cloudflare

Asks for the account ID and token, checks them with Cloudflare, and writes them to $RAPTOR_HOME/.env with 0600 permissions — not to the working directory, so they cannot be committed by accident. It needs an interactive terminal; without one (in a script, or for an AI agent) it exits 2 and says to put CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN in that file instead.

raptor version

Prints the version and the optional features this build has, such as features: server.

raptor on its own

In a terminal, raptor with no command shows the home screen: the four commands to start with and the last five crawls you ran, each with its grade, score, page count, how long ago it ran and the path of its report. The list comes from $RAPTOR_HOME/history.jsonl, which every crawl adds a line to. In a pipe or for an agent it prints the help instead.

Environment variables