MCP server and REST API

raptor serve --mcp     # MCP over stdio — the primary surface
raptor serve           # REST API on 127.0.0.1:3001

MCP (the Model Context Protocol) is how an AI assistant such as Claude calls Raptor's tools. Both transports share one SQLite job store under RAPTOR_HOME (default ~/.raptor), so a job started over MCP is visible over REST and vice versa.

The MCP tools

Ten tools, listed here in the order tools/list gives them. The crawl ones are asynchronous: raptor_crawl returns a job_id straight away, and you poll raptor_status until the job is finished before reading results.

A build without the cloudflare feature does not list raptor_screenshot, and raptor_inspect offers only its static mode, because those are the ones that build can run. Calling raptor_screenshot anyway still answers with a message saying it needs the cloudflare feature.

A typical session

raptor_crawl    { "url": "https://example.com/", "max_pages": 200 }   -> job_id
raptor_status   { "job_id": "..." }                    -> queued | running | completed | failed | cancelled
raptor_summary  { "job_id": "..." }                    -> the verdict and the top fixes
raptor_findings { "job_id": "..." }                    -> every finding, ranked
raptor_findings { "job_id": "...", "check_id": "page_titles.duplicate" }   -> one fix brief

A crawl that reaches no page (DNS, TLS, refused, timed out, blocked by robots.txt) or whose every page answered with an HTTP error ends failed, with the reason in error.

raptor_findings without a check_id gives one short row per finding: its priority, its title, how widely it reaches (for example "Every page — fix once" or "/blog/ pages") and its check id. It filters by priority (fix_now, high, medium, low, note) and by area, the part of the search journey it belongs to (reached, indexed, unique, found, described, fast_safe). With a check_id it returns that finding in full: what was found, why it matters, how to fix it, up to 25 affected URLs with the true total, and brief, a markdown fix brief ready to paste into a task or another agent. A check_id that did not fire says why: the check ran and passed, it could not run on this crawl (it needs PageSpeed data, JavaScript rendering, the site's sitemaps or its robots.txt), or it is not on this crawl, followed by the ids that did fire.

raptor_compare against a previous job_id is the one to reach for on a re-audit: it answers "what changed" directly instead of making the model diff two summaries itself. It matches findings by category, check and page, so two checks that share a name in different categories (a duplicate title and a duplicate meta description) are counted apart. Each regressions[] entry lists its new_checks by name and its new_findings as {category, check} pairs, and improvements counts resolved findings by their own category.

Every comparison reports scoring_model ({baseline, current}) and comparable. comparable is false, and comparability_warning says why, when:

• the two crawls were scored by different models: the score and issue changes then include changes in how Raptor measures, not only changes to the site. A crawl stored before Raptor recorded its model reads as model 1. Re-crawl the baseline with the current version for a like-for-like comparison. See scoring.md.
• either crawl is partial: it was cancelled, or it stopped at its page limit. Every finding on a page that crawl never fetched would otherwise count as fixed or new, so the warning names the side ("The current crawl was interrupted after 12 pages, …").

Cancelling a crawl

raptor_cancel { "job_id": "..." } stops a crawl.

• A queued job never starts: "Cancelled before it started. Nothing was fetched."
• A running crawl in this server stops fetching new pages (stopped: true). The pages it already fetched are analysed and saved, and the job stays cancelled. raptor_summary and raptor_findings then work on it and say it was interrupted, with crawl_metadata.interrupted: true in its crawl.json. PageSpeed is skipped for it.
• For a few seconds after the cancel, while those pages are analysed and saved, raptor_status reports stopping: true, and the other job tools answer "Job … is stopping: the pages it fetched are being analysed." Poll raptor_status until stopping is false: a job is finished only when its status is final and stopping is false.
• A crawl that another Raptor process runs (one sharing this RAPTOR_HOME) cannot be stopped from here. It is marked cancelled, the tool says this server cannot stop it, and that process finishes and saves its results. Until it does, raptor_status adds saving_elsewhere: true and says no pages are saved yet, the other job tools say the same, and the crawl panel keeps watching for up to ten minutes.
• REST DELETE /api/v1/jobs/{id} cancels through the same path before deleting the job.

raptor_status is read-only. Cancel used to be raptor_status {action: "cancel"}; that call now returns cancel moved to raptor_cancel: call raptor_cancel with this job_id and cancels nothing.

What the model reads

Every tool's text block is a written digest in plain words: the verdict first, then the ranked fixes in the check catalogue's own words, lists capped with their true totals, and the next call to make. A host that passes the model only text (and Claude Desktop, which can drop structured data) still gives it everything it needs to keep working.

• raptor_summary, raptor_findings and raptor_crawl also carry the same text in structuredContent.digest, because Claude Code's model reads only structuredContent when both are present.
• Every crawled string in a digest (a page title, a URL, a detail, a culprit) is flattened onto one line and cut to length, so a hostile page cannot start a line of its own or open a code span.
• Digests always use plain wording. The panels speak in the configured brand voice.
• A client on a protocol revision older than 2025-06-18 cannot read structuredContent, so it gets the full JSON as text instead, exactly as before.

raptor_summary keeps every key it had and adds: digest, score_sentence, fix_first (the top fixes as cards: findings on exactly the same pages are fixed together, so they share a card), fix_first_total, findings_by_priority, to_fix, notes, limits, report_path (the job's report.html), brand (only when brand.toml was refused) and panel. Its size does not grow with the crawl: a 1,000-page summary is within 10% of a 50-page one.

Responses are bounded

This is the part that matters most for agent use, and the part most likely to bite you if you assume otherwise.

A site-wide check affects every page, so an unbounded list grows with the crawl. Before bounding, a single default call on a 500-page crawl returned well over 100k tokens.

• raptor_query lists at most max_urls_per_issue URLs per issue (default 5, max 500), and always reports affected_url_count with the true total plus an affected_urls_truncated flag, and affected_page_count with the number of distinct URLs among the rows (each URL once; not resolved through redirects). Narrow with url_pattern rather than raising the cap. A filtered query also reports the filters it applied, in filters.
• raptor_findings lists one short row per finding, and gives one finding's detail only when you ask for it by check_id, with at most 25 affected URLs and the true total beside them.
• raptor_link_graph bounds every list by top_n, including orphans and silos, and reports the true totals beside them.
• raptor_compare bounds its regression list.
• Every result a panel reads stays within 120,000 bytes, even on a crawl where every check fires on long URLs. Claude swaps a larger result for a file pointer, which a panel cannot read.
• raptor_inspect's title, meta_description and h1s keep their full values. The page panel reads its own capped copy, page_panel: a title or description over 1,000 characters is cut with "…", and only the first 20 H1s are kept, each cut at 300 characters. title_chars, meta_description_chars and h1_count give the true sizes. Because the existing keys are never cut, a page with a pathological title or H1 list can return more than 120 KB. page_findings lists every finding on the page; when explaining them all would not fit, the first ones keep their explanations and page_findings_trimmed is true.
• resources/read is windowed: 100 KB per read by default (1 MB maximum), with a truncation notice carrying byte counts, totalBytes/truncated metadata, and a nextUri that pages through the rest via ?offset=&limit=. resources/list advertises each artifact's size so a client can decide before reading.

Field naming

affected_urls means a list everywhere. affected_url_count means a number everywhere. These were briefly inconsistent between raptor_summary and raptor_query, which read as an empty list to a caller that had seen the other tool — if you have client code written against the older shape, raptor_summary now returns affected_url_count.

Counting units

Raptor counts in four units. The field names say which one you are reading, with one exception: affected_page_count counts URLs, not pages.

worst_pages[].issue_count in raptor_summary is the number of findings on that page, not its URL rows.

culprits appears only when a finding names the offending values: on the issue for a site-wide finding (such as the AI crawlers robots.txt blocks), and on an affected_urls[] entry for one page (such as the render-blocking script URLs). An issue or a URL without culprits has no culprits key.

raptor_query's category filter takes the values its schema lists, including javascript, which finds the JavaScript checks (their category reads java_script in the results, and their check_id reads javascript.…). Each raptor_query issue also carries check_id, title, count, unit, priority and priority_label.

Argument validation

Arguments outside a tool's published inputSchema are rejected, not ignored, and the error names the offending argument alongside the accepted set. A misspelled maxpages used to be silently dropped and the crawl ran to the 500-page default — against someone's production site. Integral floats (3.0) are accepted where an integer is expected; genuinely out-of-range values are refused with a message naming the parameter.

Hints

Every tool has a title and all four behaviour hints, written out, because MCP assumes a tool with no hints may destroy data and reach the outside world. Hosts use the hints to decide when to ask you first: a read-only tool can run without a prompt, and a destructive one always asks.

"Open world" means the tool fetches the web; the others read only what Raptor stored. raptor_status is read-only so a progress panel can poll it every two seconds without a prompt each time.

Panels (MCP Apps)

MCP Apps is an extension of MCP that lets a host show a small interactive view, a panel, beside a tool's result. Where the host supports it, four of Raptor's tools come with a panel:

Which hosts show them, for a local raptor serve --mcp (stdio):

• Claude Desktop shows them, inline and fullscreen.
• VS Code (GitHub Copilot), Cursor and Goose show them inline. VS Code and Cursor offer no fullscreen, so the findings and page panels show more rows inline instead of an Expand button.
• Claude Code shows text only. Its model gets the same facts from the digests.
• claude.ai on the web, Claude on mobile and ChatGPT reach only remote servers, which need a public HTTPS address; a local stdio server does not reach them.

How they work:

• Addresses. Each panel is a resource at ui://raptor/v{version}/{panel}.html (health, findings, page, crawl). Hosts cache panels by address, so the version in the address makes an upgrade show the new panel.
• Negotiation. The server declares the io.modelcontextprotocol/ui extension to every client. A host that says at initialize that it renders text/html;profile=mcp-app sees the four panels in resources/list; a text-only client does not, so its resource picker stays clean. resources/read serves a panel to any client, whole. There is no ui.domain: a stdio server has no URL for a host to derive one from.
• Tools carry their panel in _meta.ui.resourceUri (and the older ui/resourceUri). Hosts without MCP Apps ignore it.
• What a panel may call. _meta.ui.visibility lets a panel call only raptor_status, raptor_cancel, raptor_summary, raptor_findings and raptor_inspect; hosts refuse a panel's call to raptor_crawl, raptor_query, raptor_compare, raptor_link_graph or raptor_screenshot. Each panel also refuses, in its own code, any tool beyond the ones it needs.
• No network. Each panel is one self-contained HTML document: its styles, script, icons and mascot are inline, and it makes no request of its own. Data reaches it only through the host.
• Crawled text stays text. A panel writes every crawled string as plain text, never as markup, so a hostile page title shows as its literal characters and runs nothing.
• When the host strips the data. Some hosts deliver a panel its tool's result without structuredContent (Claude Desktop, ext-apps #696). The panel then reads JSON text if there is any, or calls its own read tool once with the tool's arguments, or says plainly that no data came. It never sits blank.
• Messages to the conversation use fixed sentences with only a validated job id, check id, status word, score or grade letter filled in, never crawled text. The crawl panel tells the model once when a crawl it watched finishes, and a panel scrolled back into view later does not repeat it.
• Theme. Panels follow the host's light or dark theme and its colour variables, with Raptor's own colours as the fallback. Brand green and the priority colours are always Raptor's.
• Fonts. Panels use the host's font variables and load no font. Claude names its font as "Anthropic Sans" but delivers it through a stylesheet a panel would have to fetch, so in Claude the panels render in the browser's generic sans-serif.

Previewing the panels

The panels can be checked and screenshotted in a fake host, without a real MCP client. DIR is any folder you choose:

# After any change to templates/panels/ (seconds): write each panel's HTML.
RAPTOR_PANEL_SCENARIOS=DIR cargo test -p raptor-server --test mcp_panels write_panel_html -- --ignored
# After any change to tool output (crawls the test fixtures; a minute): record the scenarios.
RAPTOR_PANEL_SCENARIOS=DIR cargo test -p raptor-cli --test cli mcp::scenarios -- --ignored --nocapture
# Run every check and take the screenshots (needs Node; fetches Playwright through npx).
scripts/screenshot-panels.sh DIR OUT_DIR [--only health|findings|page|crawl]

Only the first command needs re-running after a panel edit. The script writes OUT_DIR/panels/*.png and OUT_DIR/checks.json, and exits 1 when a check fails.

Live progress

While a crawl runs, raptor_status reports progress for it, updated about four times a second:

• pages_crawled, pages_queued and estimated_total, with the meanings they always had (null until the first page is fetched);
• status_counts (2xx, 3xx, 4xx, 5xx), errors (fetches that got no answer, which the panel and the digest call failed fetches) and recent_errors (the last three, with their url, kind and label);
• current_url, the page fetched last;
• phase and phase_label, the phase in flight ("Fetching pages", "Reading sitemaps", …) in the same words the CLI uses, and phases_done with each finished phase's duration;
• max_pages, bar_total and bar_label for a progress bar that never promises more than the page limit ("120 of up to 500 pages");
• facts_line and tally_line, the two lines the crawl panel prints;
• interrupted, and seq, which counts the writes.

A finished job's result_summary adds grade_verdict, graded, findings_count, to_fix, duration_ms, stopped_at_page_limit, urls_not_crawled and interrupted. A failed job's holds only failure: what went wrong, in a form the crawl panel can word.

Limits

Every digest lists the crawl's limits, the things that change how far you can trust it: a crawl that stopped at its page limit or was interrupted, URLs robots.txt kept it out of, pages that could not be fetched, pages that look built by JavaScript, data saved without its pages, a score from an older scoring model, and a score that stopped at 0. It is the same ordered list the report and the CLI show. The health panel shows the first of them and says how many more there are.

When the deductions add up to more than 100, the score stops at 0, and a finding's points then say what it costs rather than how much fixing it alone would lift the score. That explanation sits beside the points wherever they are printed.

Branding

The server reads brand.toml once at start (from RAPTOR_BRAND, or $RAPTOR_HOME/brand.toml). If the file is refused, the server still starts, with the Raptor brand, and raptor_crawl and raptor_summary say so in their text and in brand.error, because Claude Desktop hides the server's own log. The panels use the brand's voice and mascot setting, but no logo, accent or agency name. See report.md.

Protocol details

• Version negotiation on initialize; only implemented capabilities are advertised, plus the io.modelcontextprotocol/ui extension (see Panels).
• Requests are handled concurrently, so responses may arrive out of request order — correlate by id, as JSON-RPC intends. Each response is one complete newline-delimited line.
• notifications/cancelled is enforced: in-flight work is aborted and, per spec, no response is sent for the cancelled request. initialize is never cancellable.
• A tool's result is returned as structuredContent, with its written digest as the text block; clients older than 2025-06-18 get the JSON as text instead.
• In MCP mode all logging goes to stderr — stdout is reserved for JSON-RPC. The log is plain text, with no colour codes, and carries no REST API-key warning, which does not apply to stdio.

REST API

Creating a job — note that crawl settings live under options, and unknown top-level fields are rejected rather than ignored:

curl -X POST http://127.0.0.1:3001/api/v1/jobs \
  -H 'Content-Type: application/json' \
  -d '{"type":"crawl","url":"https://example.com","options":{"max_pages":50}}'

crawl is the only executable job type. The others are refused with a 4xx naming what is supported, rather than accepted and left queued forever.

A served report.html runs under a locked-down content security policy: no script, no connection and no remote source, with only its embedded data: images and fonts allowed.

Webhooks

Registered webhooks fire on terminal job states: job.queued, job.completed, job.failed, job.cancelled. Cancelling a running crawl delivers job.cancelled immediately, while the crawl stops and saves the pages it fetched, and again when it has. One-shot webhooks remain exactly-once.

Security

• Auth is off unless a key is set. --api-key or RAPTOR_API_KEY. Without one, every route except /health is open to anything that can reach the port. The default bind is 127.0.0.1.
• CORS is closed by default. Allow specific origins with RAPTOR_CORS_ORIGINS.
• SSRF protection on crawl targets and webhook URLs: private, loopback, link-local and cloud-metadata ranges are refused, including IPv4 tunnelled inside IPv6, and the address vetted is the one actually dialled. Override for intranet audits with RAPTOR_ALLOW_PRIVATE_TARGETS=1.
• Request bodies are capped at 1 MB; artifact downloads are path-traversal safe.

Running more than one process

Every Raptor process sharing a RAPTOR_HOME shares the job store. Jobs carry an owner and a heartbeat, so an MCP server starting up no longer reclaims a REST server's running crawls. A process can stop only the crawls it runs itself (see Cancelling a crawl).

The trade-off: a job orphaned by a crashed process is reclaimed after 120–180 seconds, not instantly. If you want full isolation — separate job stores per client, say — point each process at its own RAPTOR_HOME.