# CLI commands

Every command is `raptor <command>`. Run `raptor <command> --help` for the exact flag list; this
document explains what each command *does* and how it works, which the help text does not.

| Command | Needs network | Input | Purpose |
|---|---|---|---|
| [`crawl`](#raptor-crawl) | yes | a URL | Crawl a site and analyse it |
| [`link`](#raptor-link) | no | `crawl.json` | Internal link graph: PageRank, orphans, hubs, silos |
| [`sitemap`](#raptor-sitemap) | no | `crawl.json` | XML sitemap of the indexable pages |
| [`inspect`](#raptor-inspect) | yes | a URL | Single-page audit, no crawl needed |
| [`serve`](#raptor-serve) | yes | — | REST API and/or MCP server |
| [`screenshot`](#raptor-screenshot) | yes | a URL | Full-page screenshot (Cloudflare feature) |
| [`extract`](#raptor-extract) | yes | a URL | AI structured extraction (Cloudflare feature) |
| [`setup`](#raptor-setup) | no | — | Store integration credentials |
| [`version`](#raptor-version) | no | — | The version and the features this build has |

Run `raptor` on its own in a terminal for the home screen: the commands to start with and your
recent crawls.

---

## Output: rich, plain and JSON

Raptor prints for whoever is reading. In an interactive terminal window (a "TTY") you get the
**rich** layout: colour, a live progress region while the crawl runs, and a summary that leads with
the grade and what to fix first. Through a pipe (`raptor crawl … | tee log.txt`), in CI, or when an
AI agent runs it, you get the **plain** layout: one `key: value` fact per line, no colour, no
progress animation, full URLs and absolute file paths, with the verdict first and the files last.
`--json` prints a small JSON summary instead, for scripts and agents that parse output.

Progress and logs always go to stderr, and results to stdout, so `raptor crawl … > summary.txt`
captures the summary while the progress stays on screen.

Each output stream decides on its own, in this order (the first rule that applies wins):

| Order | Setting | Effect |
|---|---|---|
| 1 | `--json` (crawl and inspect) | stdout prints the JSON summary |
| 2 | `--format rich` or `--format plain` | that layout |
| 3 | `--color always`, `FORCE_COLOR` set, or `CLICOLOR_FORCE` set (not `0`) | rich, in colour |
| 4 | An AI agent is detected (below) | plain |
| 5 | The stream is a TTY and `TERM` is not `dumb` | rich |
| 6 | Otherwise | plain |

Colour in the rich layout follows the usual standards: `--color always|never` first, then
`NO_COLOR` (set to anything turns colour off), `FORCE_COLOR`, and `CLICOLOR=0`. `NO_COLOR` on its
own keeps the rich layout, just without colour. Plain and JSON output never carry colour or
terminal control codes.

An **AI agent** is detected when `CLAUDE_CODE_CHILD_SESSION=1`, when `CLAUDECODE=1` and stdout is
not a TTY (an IDE terminal sets `CLAUDECODE=1` on a real terminal, where a person reads), or when
`AI_AGENT`, `AGENT` or `CURSOR_AGENT` is set. An agent gets plain output even on a terminal.

`--format` goes after `crawl` or `inspect` (`raptor crawl https://example.com --format plain`), or
before any subcommand (`raptor --format plain link crawl.json`). `raptor screenshot --format` is
still the image format, `png` or `jpeg`.

`-q` (`--quiet`) prints only errors: no progress, no summary and no log warnings, though a crawl
that has nothing to grade still names its cause on stderr; the files are still written. `-v` adds
a line per page and Raptor's own debug log; `-vv` logs every library. `--color`, `-q` and `-v`
work before or after any subcommand.

## Exit codes

An exit code is the number a command hands back to the shell or CI when it ends; `0` means
success. Every Raptor command uses the same table:

| Code | Meaning |
|---|---|
| `0` | it worked (findings never fail a run on their own; stopping at `--max-pages` is still 0) |
| `1` | runtime error: an unwritable output path, a request that failed, a backend this build lacks, a brand file Raptor cannot use |
| `2` | usage error: an unknown flag or value, or a URL without `https://` |
| `3` | nothing to grade: the start URL could not be reached, robots.txt disallows it, every page answered with an HTTP error, or the crawl ran but nothing could be graded (a bot wall, or an app shell with no readable content) |
| `4` | a quality gate failed (`--min-score`, `--fail-on`) |
| `130` | interrupted with Ctrl-C (what was crawled is saved) |

---

## `raptor crawl`

```bash
raptor crawl https://example.com/ -o crawl.json
```

Crawls a site breadth-first from the seed URL, then runs every analyser over the collected pages.
This is the command the others consume.

### How it works

**The seed is resolved first.** Before anything else, Raptor follows the seed's own redirect and
treats where it *lands* as the site under audit. Seeding `https://www.example.com/` on a site that
canonicalises to the apex used to classify every discovered link as external and end the crawl
after one page — with a passing grade and no warning. The URL that finally served a page is that
page's identity throughout; the URL as requested is preserved in its redirect chain.

**Scope** is the seed's host and port. A link to a different host is external and is recorded but
not crawled. `--follow-subdomains` widens this to subdomains of the same registrable domain.
Scheme is deliberately *not* compared, so an `http://` link on an HTTPS site is still internal.

**Politeness** is a single shared gate, not a per-worker sleep: `--concurrency` workers share one
`--delay` interval, so three workers with a 250 ms delay produce one request per 250 ms, not three.
`robots.txt` is fetched and honoured (RFC 9309), and a `Crawl-delay` larger than `--delay` wins.
A robots.txt that answers 5xx or 429 makes every path disallowed, as Google does (RFC 9309 says so
for 5xx only). A robots.txt that cannot be fetched at all (DNS, refused, TLS, timeout) does not
block the crawl: Raptor crawls without rules and lets the start page decide whether the site is
reachable. That is a deliberate departure from RFC 9309 §2.3.1.4, and Google treats an unreachable
robots.txt as blocking the whole site, so the robots line says so.
`--ignore-robots` exists and is labelled *not recommended* for good reason: it is for auditing
sites you control.

**Budget.** `--max-pages` is an exact upper bound, not approximate. `--depth` bounds distance from
the seed. A crawl that reaches `--max-pages` with URLs still waiting prints "Stopped at the page
limit" with how many it left behind, and the JSON records them as `stopped_at_page_limit` and
`urls_not_crawled`.

**Safety.** Crawl targets are checked against private, loopback, link-local and cloud-metadata
ranges to prevent SSRF, with the DNS resolution that is vetted being the one actually dialled.
Set `RAPTOR_ALLOW_PRIVATE_TARGETS=1` to audit an intranet or a local fixture.

### What it prints

In a terminal the crawl opens with a header (the `///` mark, the URL and the settings on one quiet
line) and a live progress region: the phase, pages done and queued, a tally of answers by status,
the rate and the page in flight. Each finished phase collapses into one ✓ line with its time, and
robots.txt and fetch errors print as they happen (the first ten errors, then one line saying the
summary counts the rest).

The end of the run leads with the grade and the score, one sentence on where the points went, and
**Fix first**: the five most important things to fix, in plain words, each with how far it reaches,
the points it is costing, how to fix it, an example page and the stable check id (such as
`links.broken_internal`). Below that come the run's limits, any quality gates, the files written
and what to do next.

Piped, the same facts come one per line:

```
raptor 0.3.0 crawl https://example.com/
result: ok (exit 0)
crawl: Crawl complete: 125 pages in 31s.
headline: 83/100: Good. Here is what is holding the score back.
grade: B 83/100 (Good)
why: Render-blocking script on every page costs the most, about 5.7 points.
findings: 21 to fix, 2 notes; 96 checks passed
fix first:
1. High · javascript.js_render_blocking · 125 pages wait for a script before showing anything
   reach: Every page — fix once; points: +5.7
   fix: Add defer to the script tag (or async if no other script depends on it), …
   example: https://example.com/ (https://example.com/assets/app.js)
…
next: Map internal links: raptor link /home/me/crawl-example.com-2026-10-17.json
files:
json: /home/me/crawl-example.com-2026-10-17.json
html: /home/me/crawl-example.com-2026-10-17.html
```

`--json` prints the same summary as one line of JSON (pretty-printed in a terminal), with the schema
name `raptor.crawl-summary/1`: the result and exit code, the headline, the score, the counts, the
Fix first items with their copy, the limits, the gates, the next steps and the absolute paths of the
files. It never lists every affected URL, so it stays small on a large crawl; the full detail is in
the crawl JSON file.

### Output files

By default writes the JSON *and* a self-contained HTML report beside it, sharing the stem of
`--output`: `-o crawl.json` produces `crawl.json`, `crawl.html`, and `crawl.csv` with `--csv`.

| Flag | Effect |
|---|---|
| `--no-html` | JSON only — what you want when an agent or a script is the consumer |
| `--csv` | Also write a CSV, one row per page |
| `--summary-only` | Metadata, issues and summary; drops the per-page array |
| `--include-body-text` | Include page text in the JSON (excluded by default; it dominates file size) |
| `--json-compact` | Minified JSON |
| `--json` | Print the JSON summary on stdout instead of text (the crawl JSON file is still written) |

The HTML report is genuinely self-contained: no CDN, no external fonts, nothing fetched when it is
opened. It is safe to email to a client. The report embeds every crawled page; only per-URL detail
strings are capped (100 per finding), and the report says so.

### Branding

| Flag or variable | Effect |
|---|---|
| `--brand <FILE>` | Put your agency's name, logo and colour on the HTML report, from a `brand.toml`. See [report.md](report.md#customising-the-report) |
| `RAPTOR_BRAND` | The path of a `brand.toml` to use when `--brand` is not given |
| `--no-brand` | Use the plain Raptor brand, ignoring `--brand`, `RAPTOR_BRAND` and `brand.toml` in Raptor's home folder |

A brand file can also set the voice: with `voice = "plain"` the terminal's rich output speaks
plainly too.

### Quality gates

Gates let CI fail a build when the site gets worse. A run with a failed gate still writes its files
and exits 4.

| Flag | Fails the run when |
|---|---|
| `--min-score N` | the score is below `N` (0 to 100) |
| `--fail-on PRIORITY` | any finding is at that priority or more urgent: `fix-now`, `high`, `medium` or `low` |

Notes never fail a run (`--fail-on note` is refused). A crawl with nothing to grade exits 3 before
any gate is checked, so a blocked or unreachable site never passes by accident.

```bash
raptor crawl https://staging.example.com --json --min-score 80 --fail-on high > summary.json
```

### Stopping a crawl

Press Ctrl-C once to stop: Raptor lets the requests already in flight finish, analyses the pages it
has, writes the files and exits 130. The summary says the crawl was interrupted and gives the
command that runs it again. What one press can stop depends on when it lands:

- while pages are still being fetched, the crawl stops there and keeps what it has;
- once every page is fetched, it skips the sitemap check (the summary says so, exit 130);
- during `--pagespeed`, it skips the PageSpeed tests and saves the crawl as it is (exit 130);
- while the crawl is analysed or saved, nothing is left to stop: the crawl is saved whole and the
  run ends with its usual result.

Press Ctrl-C a second time to quit at once without saving.

### When a crawl cannot see the site

When Raptor gets nothing it can grade, the summary names the cause instead of a grade:

- the start URL could not be reached, and why: the name did not resolve (DNS), the secure
  connection failed (TLS), the connection was refused, it timed out, the connection closed before a
  response, or the address is private and blocked;
- robots.txt disallows the start URL, or robots.txt answered 5xx or 429, so every path is
  disallowed;
- every page Raptor fetched answered with an HTTP error, counted by status, as in
  `HTTP 403 on 1 of 1`.

```
result: nothing to grade (exit 3)
failure: Could not reach https://example.com/: connection refused (…). Nothing was crawled.
```

Only the JSON is written, because it records what failed. `crawl_metadata.seed_error` holds the
start URL's failure (`url`, `kind` such as `dns`, `tls`, `timeout` or `refused`, and `detail`), and
`crawl_metadata.robots` holds what robots.txt said. There is no HTML report and no CSV. If an
earlier run left them at the paths this run would have used, they are removed, and a
`cleanup: Removed …` line names each one, so an old report never sits beside a crawl that reached
nothing. The run exits 3. With `-q` the cause still prints, on stderr.

### Rendering and Core Web Vitals

`--backend playwright` renders pages in local Chrome instead of fetching raw HTML — needed for
sites that inject their content, title or canonical client-side. Requires the `playwright` feature.
The release binaries are built without it, so build Raptor from source with
`cargo build --release --features playwright` to get it.
`--pagespeed` adds Core Web Vitals via the Google PageSpeed API (`--pagespeed-key`, or
`PAGESPEED_API_KEY`); `--pagespeed-sample N` limits it to N pages, since the API is rate-limited
and one call per page on a large crawl is slow.

---

## `raptor link`

```bash
raptor link crawl.json            # terminal summary
raptor link crawl.json --json     # full report as JSON
```

Builds the internal link graph from a completed crawl and reports PageRank, orphan pages, hubs,
silos and connected components. It performs no network I/O.

See [link-graph.md](link-graph.md) for what each number means and how it is computed — that
document is the one to read before acting on the output.

`--top N` bounds every ranked list (default 15). The report always carries the true totals
(`orphan_page_count`, `silo_count`) alongside the truncated lists.

---

## `raptor sitemap`

```bash
raptor sitemap crawl.json -o sitemap.xml
```

Writes an XML sitemap containing the crawl's **indexable** pages: 2xx HTML that is not `noindex`.
A 404, a redirect, a PDF or a `noindex` page is excluded.

If nothing qualifies — or the crawl was run with `--summary-only`, so there is no page array to
read — it writes nothing and exits non-zero, saying which of the two applies. It will not produce
a schema-valid-looking empty `<urlset>`, which Search Console reads as an empty sitemap.

Above the 50,000-URL protocol limit it emits a `<sitemapindex>` plus chunk files rather than one
oversized file, which Google rejects whole.

---

## `raptor inspect`

```bash
raptor inspect https://example.com/pricing
```

Audits one URL and prints a single-page report: title, meta description, canonical, indexability
with the directives that decided it, status, word count, readability, links, images, schema types,
security headers, and the findings that apply to that page, most important first and in the same
plain words as a crawl, with Notes last.

A page that answers with an error (a 404, say) is still inspected, and the report opens by saying
it is gone. A page that never answers exits 3, with the same message a crawl gives for an
unreachable start URL. `--json` prints a JSON summary (schema `raptor.inspect-summary/1`) instead of
text, and `-o page.json` still saves the full result, in its old shape.

Useful for checking one page quickly, and for confirming a finding from a crawl without re-running
it. Inspecting a URL that redirects reports on the page that served it and still tells you about
the redirect.

With the `cloudflare` feature: `--rendered` audits the JS-rendered DOM, and `--compare` shows
static vs rendered side by side — the fastest way to see what a crawler misses on a JS-heavy site.

---

## `raptor serve`

```bash
raptor serve                 # REST API on 127.0.0.1:3001
raptor serve --mcp           # MCP server over stdio
```

See [mcp.md](mcp.md) for the tools, the job lifecycle and the REST endpoints.

Two things worth knowing here:

- **Auth is off unless you set a key.** `--api-key`, or `RAPTOR_API_KEY` in the environment. With
  no key, every route except `/api/v1/health` is open to anything that can reach the port — which
  is why the default bind is `127.0.0.1`, not `0.0.0.0`.
- **Jobs are shared state.** Every Raptor process on the same `RAPTOR_HOME` (default `~/.raptor`)
  reads the same SQLite job store. Running an MCP server and a REST server at once is supported;
  jobs are owned and heartbeated, so one process no longer reclaims another's running work.

---

## `raptor screenshot`

Full-page or element screenshots through Cloudflare Browser Rendering. Requires the `cloudflare`
feature and credentials. `--full-page`, `--selector`, `--viewport WxH`, `--format png|jpeg`.

## `raptor extract`

AI structured extraction from a page through Cloudflare, with an SEO preset by default or your own
`--prompt` / `--schema`. Requires the `cloudflare` feature and credentials.

## `raptor setup`

```bash
raptor setup cloudflare
```

Asks for the account ID and token, checks them with Cloudflare, and writes them to
`$RAPTOR_HOME/.env` with `0600` permissions — not to the working directory, so they cannot be
committed by accident. It needs an interactive terminal; without one (in a script, or for an AI
agent) it exits 2 and says to put `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` in that file
instead.

## `raptor version`

Prints the version and the optional features this build has, such as `features: server`.

## `raptor` on its own

In a terminal, `raptor` with no command shows the home screen: the four commands to start with and
the last five crawls you ran, each with its grade, score, page count, how long ago it ran and the
path of its report. The list comes from `$RAPTOR_HOME/history.jsonl`, which every crawl adds a line
to. In a pipe or for an agent it prints the help instead.

---

## Environment variables

| Variable | Purpose |
|---|---|
| `RAPTOR_HOME` | Data directory (default `~/.raptor`): job database, job output, credentials |
| `RAPTOR_API_KEY` | Bearer token required by the REST API and MCP HTTP routes |
| `RAPTOR_ALLOW_PRIVATE_TARGETS` | Allow crawling private/loopback addresses (intranet audits, local fixtures) |
| `RAPTOR_CORS_ORIGINS` | Comma-separated CORS allowlist; CORS is closed by default |
| `RAPTOR_MAX_CONCURRENT_CRAWLS` | Concurrent crawl cap for the server (default 3) |
| `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN` | Cloudflare rendering credentials |
| `PAGESPEED_API_KEY` | Google PageSpeed API key |
| `RAPTOR_BRAND` | The `brand.toml` the HTML report uses when `--brand` is not given |
| `RUST_LOG` | Log filter, which wins over `-q` and `-v` (for example `RUST_LOG=raptor_core::crawler=debug`). Without it a crawl logs warnings, `-v` adds Raptor's own debug lines and `-vv` every library's. In MCP mode logs go to stderr, never stdout |
| `NO_COLOR`, `FORCE_COLOR`, `CLICOLOR`, `CLICOLOR_FORCE` | Colour standards; see [Output](#output-rich-plain-and-json) |
| `CLAUDE_CODE_CHILD_SESSION`, `CLAUDECODE`, `AI_AGENT`, `AGENT`, `CURSOR_AGENT` | Tell Raptor an AI agent is reading, which gets plain output; see [Output](#output-rich-plain-and-json) |
| `COLORTERM` | `truecolor` or `24bit` paints the `///` mark in Raptor's exact green |
| `COLORFGBG` | A light background (`…;15` or `…;7`) switches the mark to the darker green |