Open to argument

Thanks for looking. Raptor is an SEO audit tool, which means its whole value is that its output is true. A tool that confidently reports a problem that isn't there wastes someone's afternoon and loses their trust permanently. Most of the guidance below follows from that.

Getting set up

cargo build --release
cargo test --workspace

You need a stable Rust toolchain (edition 2021, rustc ≥ 1.88). No other services are required — the default build reaches nothing external, and the test suite runs offline.

Before opening a pull request:

cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all --check

If you touched anything feature-gated, check it still builds both ways:

cargo check --workspace --all-targets --no-default-features
cargo check --workspace --all-targets --features cloudflare
cargo check --workspace --all-targets --features playwright

The most valuable bug report

False positives. A check that fires on a page that is actually fine is worse than a check that doesn't exist. Every one found so far came from someone running Raptor against a real site and not believing the output — for example, three pages reported as having no <title> that all plainly had one, because invalid markup in <head> had moved the title into <body>.

If you hit one, the most useful report includes the URL (or a snippet of markup that reproduces it), what Raptor said, and what's actually true. A failing test case is even better.

Writing checks and fixes

Verify against a real fixture, not by reading the code. Stand up a local site (the raptor-fixtures crate serves them; see Fixtures, CLI tests and the gallery) and run the actual binary against it. RAPTOR_ALLOW_PRIVATE_TARGETS=1 is required to crawl loopback addresses.

Prove your test catches the bug. Revert your fix, confirm the new test fails, then restore it. This sounds pedantic and it is not: a "regression test" that passes against the unfixed code is worse than none, and we have shipped one. If a test asserts on a rendered report, make sure the assertion targets the specific thing you fixed — a string can appear in five places and only one of them exercises your code path.

Never weaken a test to make it pass. If a test fails after your change, either the change is wrong or the test encoded the old behaviour. Both happen. Fix the right one, and say in the PR which it was.

Find every consumer of what you changed. The recurring failure mode in this codebase has been fixing a computation and leaving its readers alone — a guard added to a library while the only caller kept using the unguarded helper, a value's meaning changed while four joins still compared the old thing. grep -rn for every reader before you call a fix done.

Say what a check cannot know. Several checks are conditional on crawl coverage — "orphan page" means "nothing in this crawl links here". Where that matters, the docs say so, and new checks should do the same rather than implying more certainty than they have.

Adding a check

Every check lives in one place, the check catalogue, and every surface reads its words from there.

1. Add an entry to raptor-core/src/catalogue/checks.toml. Put it in catalogue order: by area in journey order (reached, indexed, unique, found, described, fast and safe), then by category in registry order. The id is {category}.{check}. Give it a name, severity, impact, area, unit and the copy: titles for one and for many, what Raptor found, why it matters and how to fix it.
2. Emit it from the analyzer with catalogue::issue(crate::cid!("category.check"), rows), or catalogue::site_issue for a site-level finding. Analyzers hold no copy: per-page detail and culprits are data, and the words come from the catalogue.
3. Put any threshold in a named const in the analyzer, use it in the check, and wire it through catalogue::limits::limit_for. Copy states a threshold only through {limit}, so the words cannot drift from the rule.
4. Follow the copy rules in the catalogue's tests: plain words, jargon defined on first use, a fix that opens with what to do and where, and no exclamation marks. cargo test -p raptor-core --lib catalogue checks the mechanical ones.
5. Regenerate the docs with RAPTOR_UPDATE_DOCS=1 cargo test -p raptor-core --lib catalogue::docs, which rewrites docs/checks.md.

Fixtures, CLI tests and the gallery

The sites. raptor-fixtures/ serves the sites Raptor is tested against: clean (the control), broken, canonical-variants, js-heavy and robots-blocked, each hand-written under raptor-fixtures/sites/<name>/, plus synthetic, which is generated with an exact count of each defect. clean should grade A, but today it grades F when served locally: Raptor flags every page with http_urls and localhost_outlinks because the fixtures serve plain http from 127.0.0.1. Those are its only two findings, and it grades A once Raptor stops judging a locally served site on them. Each site's routes.json documents its pages: what the site is for, every path it serves with the file, status and headers it answers with, and, where it matters, a note on why the path is there. Read it before you change a page. A typo in a fixture shows up as a 404 or a missing header that looks exactly like a Raptor bug.

To crawl them by hand:

cargo run -p raptor-fixtures -- serve all

This prints a name=URL line for every site and for each failure a crawl can meet before it reads a page (refused, dns, tls, hang, robots-500), and serves until Ctrl-C. serve clean serves one site.

Tests that run the raptor binary go in raptor-cli/tests/cli/, as a module of that one test binary, and start the child through harness::raptor. It runs Raptor in a temp directory, allows loopback targets, and removes the shell variables that would change its output (colour, terminal width, proxies, logging, credentials, AI-agent detection), so a test behaves the same in your shell and in CI. harness::run kills a child that is still running after two minutes. Don't crawl fixtures from raptor-core's test binaries: RAPTOR_ALLOW_PRIVATE_TARGETS is process-wide there and races the SSRF-guard tests.

Snapshots. CLI output is pinned with insta snapshots, bound through harness::insta_settings() so ports, timings, dates, temp paths and the version are redacted. A new or changed snapshot fails its test, and insta writes the candidate beside it as a .snap.new file. Read it. When it is right, accept it:

INSTA_UPDATE=always cargo test -p raptor-cli --test cli

then run the tests again without the variable, which confirms they pass and clears the leftover .snap.new, and commit the .snap. The variable rewrites file snapshots only; an inline @"…" snapshot is edited by hand. CI never writes snapshots: a mismatch there just fails.

Large fixtures. The suite and the gallery use a 2,000-page synthetic site. The 20,000-page one runs only when RAPTOR_LARGE_FIXTURES=1 is set (or --large is passed to the gallery), because it is slow.

The gallery runs Raptor once against every fixture and keeps each case's command, output, exit code and report in its own directory. The screenshot script then captures every report in light, dark, 390px mobile and print, and fails if any capture made a network request:

cargo build --release -p raptor-cli
cargo run -p raptor-fixtures -- gallery --out target/gallery
scripts/screenshot-reports.sh target/gallery target/screenshots

--tty also captures each run on a terminal, and --real URL adds a real site. The screenshot script needs Node (npx fetches a pinned Playwright) and python3. Keep this output out of git: target/ is ignored, and a directory outside the repository works as well.

Code style

Match the surrounding code. A few conventions worth knowing:

• Comments explain why, not what. The code says what it does. The comment should say why it is that way — usually the bug that made it necessary, concretely. // strip the fragment is noise; explaining that fragments were becoming page identity and putting #toc into sitemaps is useful to the next person.
• Prefer a named helper over a clever one-liner when a reader would otherwise have to reconstruct the reasoning.
• Keep MCP and report output bounded. Any list that grows with crawl size needs a cap and a reported true total. A tool that returns 300k tokens is unusable to the audience Raptor is for.

Commits and pull requests

Conventional commit prefixes (fix:, feat:, docs:, refactor:, test:), with a scope where it helps (fix(crawler):). Write the body for someone who will read it in a year while bisecting: what was wrong, what the user-visible consequence was, and what changed.

Keep pull requests focused. A PR that fixes one thing and is easy to verify will be merged faster than one that fixes four.

If your change alters behaviour a user could notice, add a line to CHANGELOG.md under Unreleased, describing the effect rather than the implementation.

Reporting security issues

Please don't open a public issue. See SECURITY.md.