# MCP server and REST API ```bash raptor serve --mcp # MCP over stdio — the primary surface raptor serve # REST API on 127.0.0.1:3001 ``` MCP (the Model Context Protocol) is how an AI assistant such as Claude calls Raptor's tools. Both transports share one SQLite job store under `RAPTOR_HOME` (default `~/.raptor`), so a job started over MCP is visible over REST and vice versa. --- ## The MCP tools Ten tools, listed here in the order `tools/list` gives them. The crawl ones are asynchronous: `raptor_crawl` returns a `job_id` straight away, and you poll `raptor_status` until the job is finished before reading results. | Tool | Title | Purpose | |---|---|---| | `raptor_crawl` | Crawl a site | Start a crawl. Returns a `job_id`. | | `raptor_status` | Crawl job status | Check one job and its live progress, or list recent jobs. Read-only. | | `raptor_cancel` | Cancel a crawl | Stop a queued or running crawl. The pages it already fetched are kept. | | `raptor_summary` | Site health summary | The verdict: score, grade, what holds the score back, the top fixes and the crawl's limits. | | `raptor_findings` | Findings to fix | Every finding, ranked, most important first; with a `check_id`, one finding in full with a fix brief. | | `raptor_query` | Query crawl results | Filter results by URL pattern, category and severity; paginated. | | `raptor_inspect` | Inspect one page | Single-page audit, synchronous, no job needed. | | `raptor_compare` | Compare two crawls | Score change, new and fixed problems, and whether the two are like-for-like. | | `raptor_link_graph` | Internal link graph | PageRank, orphans, hubs, silos — see [link-graph.md](link-graph.md). | | `raptor_screenshot` | Screenshot a page | Screenshot via Cloudflare. Listed only in builds with the `cloudflare` feature. | A build without the `cloudflare` feature does not list `raptor_screenshot`, and `raptor_inspect` offers only its `static` mode, because those are the ones that build can run. Calling `raptor_screenshot` anyway still answers with a message saying it needs the `cloudflare` feature. ### A typical session ``` raptor_crawl { "url": "https://example.com/", "max_pages": 200 } -> job_id raptor_status { "job_id": "..." } -> queued | running | completed | failed | cancelled raptor_summary { "job_id": "..." } -> the verdict and the top fixes raptor_findings { "job_id": "..." } -> every finding, ranked raptor_findings { "job_id": "...", "check_id": "page_titles.duplicate" } -> one fix brief ``` A crawl that reaches no page (DNS, TLS, refused, timed out, blocked by robots.txt) or whose every page answered with an HTTP error ends `failed`, with the reason in `error`. `raptor_findings` without a `check_id` gives one short row per finding: its priority, its title, how widely it reaches (for example "Every page — fix once" or "/blog/ pages") and its check id. It filters by `priority` (`fix_now`, `high`, `medium`, `low`, `note`) and by `area`, the part of the search journey it belongs to (`reached`, `indexed`, `unique`, `found`, `described`, `fast_safe`). With a `check_id` it returns that finding in full: what was found, why it matters, how to fix it, up to 25 affected URLs with the true total, and `brief`, a markdown fix brief ready to paste into a task or another agent. A `check_id` that did not fire says why: the check ran and passed, it could not run on this crawl (it needs PageSpeed data, JavaScript rendering, the site's sitemaps or its robots.txt), or it is not on this crawl, followed by the ids that did fire. `raptor_compare` against a previous `job_id` is the one to reach for on a re-audit: it answers "what changed" directly instead of making the model diff two summaries itself. It matches findings by category, check and page, so two checks that share a name in different categories (a duplicate title and a duplicate meta description) are counted apart. Each `regressions[]` entry lists its `new_checks` by name and its `new_findings` as `{category, check}` pairs, and `improvements` counts resolved findings by their own category. Every comparison reports `scoring_model` (`{baseline, current}`) and `comparable`. `comparable` is `false`, and `comparability_warning` says why, when: - the two crawls were scored by different models: the score and issue changes then include changes in how Raptor measures, not only changes to the site. A crawl stored before Raptor recorded its model reads as model 1. Re-crawl the baseline with the current version for a like-for-like comparison. See [scoring.md](scoring.md#counting-units). - either crawl is partial: it was cancelled, or it stopped at its page limit. Every finding on a page that crawl never fetched would otherwise count as fixed or new, so the warning names the side ("The current crawl was interrupted after 12 pages, …"). --- ## Cancelling a crawl `raptor_cancel { "job_id": "..." }` stops a crawl. - A **queued** job never starts: "Cancelled before it started. Nothing was fetched." - A **running** crawl in this server stops fetching new pages (`stopped: true`). The pages it already fetched are analysed and saved, and the job stays `cancelled`. `raptor_summary` and `raptor_findings` then work on it and say it was interrupted, with `crawl_metadata.interrupted: true` in its `crawl.json`. PageSpeed is skipped for it. - For a few seconds after the cancel, while those pages are analysed and saved, `raptor_status` reports `stopping: true`, and the other job tools answer "Job … is stopping: the pages it fetched are being analysed." Poll `raptor_status` until `stopping` is `false`: a job is finished only when its status is final **and** `stopping` is `false`. - A crawl that another Raptor process runs (one sharing this `RAPTOR_HOME`) cannot be stopped from here. It is marked cancelled, the tool says this server cannot stop it, and that process finishes and saves its results. Until it does, `raptor_status` adds `saving_elsewhere: true` and says no pages are saved yet, the other job tools say the same, and the crawl panel keeps watching for up to ten minutes. - REST `DELETE /api/v1/jobs/{id}` cancels through the same path before deleting the job. `raptor_status` is read-only. Cancel used to be `raptor_status {action: "cancel"}`; that call now returns `cancel moved to raptor_cancel: call raptor_cancel with this job_id` and cancels nothing. --- ## What the model reads Every tool's text block is a **written digest** in plain words: the verdict first, then the ranked fixes in the check catalogue's own words, lists capped with their true totals, and the next call to make. A host that passes the model only text (and Claude Desktop, which can drop structured data) still gives it everything it needs to keep working. - `raptor_summary`, `raptor_findings` and `raptor_crawl` also carry the same text in `structuredContent.digest`, because Claude Code's model reads only `structuredContent` when both are present. - Every crawled string in a digest (a page title, a URL, a detail, a culprit) is flattened onto one line and cut to length, so a hostile page cannot start a line of its own or open a code span. - Digests always use plain wording. The panels speak in the configured brand voice. - A client on a protocol revision older than `2025-06-18` cannot read `structuredContent`, so it gets the full JSON as text instead, exactly as before. `raptor_summary` keeps every key it had and adds: `digest`, `score_sentence`, `fix_first` (the top fixes as cards: findings on exactly the same pages are fixed together, so they share a card), `fix_first_total`, `findings_by_priority`, `to_fix`, `notes`, `limits`, `report_path` (the job's `report.html`), `brand` (only when `brand.toml` was refused) and `panel`. Its size does not grow with the crawl: a 1,000-page summary is within 10% of a 50-page one. --- ## Responses are bounded This is the part that matters most for agent use, and the part most likely to bite you if you assume otherwise. A site-wide check affects every page, so an unbounded list grows with the crawl. Before bounding, a single default call on a 500-page crawl returned well over 100k tokens. - `raptor_query` lists at most `max_urls_per_issue` URLs per issue (default 5, max 500), and always reports `affected_url_count` with the true total plus an `affected_urls_truncated` flag, and `affected_page_count` with the number of distinct URLs among the rows (each URL once; not resolved through redirects). **Narrow with `url_pattern` rather than raising the cap.** A filtered query also reports the filters it applied, in `filters`. - `raptor_findings` lists one short row per finding, and gives one finding's detail only when you ask for it by `check_id`, with at most 25 affected URLs and the true total beside them. - `raptor_link_graph` bounds every list by `top_n`, including orphans and silos, and reports the true totals beside them. - `raptor_compare` bounds its regression list. - Every result a panel reads stays within **120,000 bytes**, even on a crawl where every check fires on long URLs. Claude swaps a larger result for a file pointer, which a panel cannot read. - `raptor_inspect`'s `title`, `meta_description` and `h1s` keep their full values. The page panel reads its own capped copy, `page_panel`: a title or description over 1,000 characters is cut with "…", and only the first 20 H1s are kept, each cut at 300 characters. `title_chars`, `meta_description_chars` and `h1_count` give the true sizes. Because the existing keys are never cut, a page with a pathological title or H1 list can return more than 120 KB. `page_findings` lists every finding on the page; when explaining them all would not fit, the first ones keep their explanations and `page_findings_trimmed` is `true`. - `resources/read` is windowed: 100 KB per read by default (1 MB maximum), with a truncation notice carrying byte counts, `totalBytes`/`truncated` metadata, and a `nextUri` that pages through the rest via `?offset=&limit=`. `resources/list` advertises each artifact's `size` so a client can decide before reading. ### Field naming `affected_urls` means **a list** everywhere. `affected_url_count` means **a number** everywhere. These were briefly inconsistent between `raptor_summary` and `raptor_query`, which read as an empty list to a caller that had seen the other tool — if you have client code written against the older shape, `raptor_summary` now returns `affected_url_count`. ### Counting units Raptor counts in four units. The field names say which one you are reading, with one exception: `affected_page_count` counts URLs, not pages. | Field | Counts | |---|---| | `issue_breakdown` (`raptor_summary`), `summary.total_issues` (`crawl.json`) | URL rows. One check can list the same page more than once. | | `affected_url_count` (`raptor_query`) | URL rows. | | `affected_page_count` (`raptor_query`) | Distinct URLs among one finding's rows, each once, exactly as written: not resolved through redirects, not limited to crawled pages. | | `findings_count` (`raptor_summary`) | Findings: checks that fired. | | `pages_with_findings` (`raptor_summary`) | Distinct crawled pages at least one finding lists, after resolving each URL to the page that served it. `null` only for a stored crawl with no page data written before Raptor recorded the count. | | `count` on a `raptor_findings` row, and a finding's `page_count` | Pages, resolved through redirects and canonical aliases. | | `count` on a `raptor_query` issue | The same shown count the report, the CLI and `raptor_findings` print: pages for a page check, URLs for a URL check, per its `unit`. Its `title` is the same one too. A URL filter narrows `affected_urls`, never `title` or `count`. | `worst_pages[].issue_count` in `raptor_summary` is the number of findings on that page, not its URL rows. `culprits` appears only when a finding names the offending values: on the issue for a site-wide finding (such as the AI crawlers robots.txt blocks), and on an `affected_urls[]` entry for one page (such as the render-blocking script URLs). An issue or a URL without culprits has no `culprits` key. `raptor_query`'s `category` filter takes the values its schema lists, including `javascript`, which finds the JavaScript checks (their `category` reads `java_script` in the results, and their `check_id` reads `javascript.…`). Each `raptor_query` issue also carries `check_id`, `title`, `count`, `unit`, `priority` and `priority_label`. ### Argument validation Arguments outside a tool's published `inputSchema` are **rejected**, not ignored, and the error names the offending argument alongside the accepted set. A misspelled `maxpages` used to be silently dropped and the crawl ran to the 500-page default — against someone's production site. Integral floats (`3.0`) are accepted where an integer is expected; genuinely out-of-range values are refused with a message naming the parameter. --- ## Hints Every tool has a `title` and all four behaviour hints, written out, because MCP assumes a tool with no hints may destroy data and reach the outside world. Hosts use the hints to decide when to ask you first: a read-only tool can run without a prompt, and a destructive one always asks. | Tool | Read-only | Destructive | Idempotent | Open world | |---|---|---|---|---| | `raptor_crawl` | no | no | no | yes | | `raptor_status` | yes | no | yes | no | | `raptor_cancel` | no | yes | yes | no | | `raptor_summary` | yes | no | yes | no | | `raptor_findings` | yes | no | yes | no | | `raptor_query` | yes | no | yes | no | | `raptor_inspect` | yes | no | yes | yes | | `raptor_compare` | yes | no | yes | no | | `raptor_link_graph` | yes | no | yes | no | | `raptor_screenshot` | yes | no | yes | yes | "Open world" means the tool fetches the web; the others read only what Raptor stored. `raptor_status` is read-only so a progress panel can poll it every two seconds without a prompt each time. --- ## Panels (MCP Apps) MCP Apps is an extension of MCP that lets a host show a small interactive view, a **panel**, beside a tool's result. Where the host supports it, four of Raptor's tools come with a panel: | Tool | Panel | |---|---| | `raptor_summary` | **Site health**: the grade, one sentence on what holds the score back, the top three fixes, and buttons to open the report and see every finding. | | `raptor_findings` | **Findings**: the ranked list inline; expanded to fullscreen, a filterable, sortable table beside one finding's fix brief, with Copy fix brief and Ask your assistant to fix. | | `raptor_inspect` | **Page check**: the page's status, title, description and headings with their lengths, and the findings on it. | | `raptor_crawl` | **Crawl progress**: a live bar, counts, the phase in flight and recent errors, polling `raptor_status`; Cancel crawl while it runs and Show summary when it is done. | Which hosts show them, for a local `raptor serve --mcp` (stdio): - **Claude Desktop** shows them, inline and fullscreen. - **VS Code (GitHub Copilot)**, **Cursor** and **Goose** show them inline. VS Code and Cursor offer no fullscreen, so the findings and page panels show more rows inline instead of an Expand button. - **Claude Code** shows text only. Its model gets the same facts from the digests. - claude.ai on the web, Claude on mobile and ChatGPT reach only remote servers, which need a public HTTPS address; a local stdio server does not reach them. How they work: - **Addresses.** Each panel is a resource at `ui://raptor/v{version}/{panel}.html` (`health`, `findings`, `page`, `crawl`). Hosts cache panels by address, so the version in the address makes an upgrade show the new panel. - **Negotiation.** The server declares the `io.modelcontextprotocol/ui` extension to every client. A host that says at `initialize` that it renders `text/html;profile=mcp-app` sees the four panels in `resources/list`; a text-only client does not, so its resource picker stays clean. `resources/read` serves a panel to any client, whole. There is no `ui.domain`: a stdio server has no URL for a host to derive one from. - **Tools carry their panel** in `_meta.ui.resourceUri` (and the older `ui/resourceUri`). Hosts without MCP Apps ignore it. - **What a panel may call.** `_meta.ui.visibility` lets a panel call only `raptor_status`, `raptor_cancel`, `raptor_summary`, `raptor_findings` and `raptor_inspect`; hosts refuse a panel's call to `raptor_crawl`, `raptor_query`, `raptor_compare`, `raptor_link_graph` or `raptor_screenshot`. Each panel also refuses, in its own code, any tool beyond the ones it needs. - **No network.** Each panel is one self-contained HTML document: its styles, script, icons and mascot are inline, and it makes no request of its own. Data reaches it only through the host. - **Crawled text stays text.** A panel writes every crawled string as plain text, never as markup, so a hostile page title shows as its literal characters and runs nothing. - **When the host strips the data.** Some hosts deliver a panel its tool's result without `structuredContent` (Claude Desktop, ext-apps #696). The panel then reads JSON text if there is any, or calls its own read tool once with the tool's arguments, or says plainly that no data came. It never sits blank. - **Messages to the conversation** use fixed sentences with only a validated job id, check id, status word, score or grade letter filled in, never crawled text. The crawl panel tells the model once when a crawl it watched finishes, and a panel scrolled back into view later does not repeat it. - **Theme.** Panels follow the host's light or dark theme and its colour variables, with Raptor's own colours as the fallback. Brand green and the priority colours are always Raptor's. - **Fonts.** Panels use the host's font variables and load no font. Claude names its font as "Anthropic Sans" but delivers it through a stylesheet a panel would have to fetch, so in Claude the panels render in the browser's generic sans-serif. ### Previewing the panels The panels can be checked and screenshotted in a fake host, without a real MCP client. `DIR` is any folder you choose: ```bash # After any change to templates/panels/ (seconds): write each panel's HTML. RAPTOR_PANEL_SCENARIOS=DIR cargo test -p raptor-server --test mcp_panels write_panel_html -- --ignored # After any change to tool output (crawls the test fixtures; a minute): record the scenarios. RAPTOR_PANEL_SCENARIOS=DIR cargo test -p raptor-cli --test cli mcp::scenarios -- --ignored --nocapture # Run every check and take the screenshots (needs Node; fetches Playwright through npx). scripts/screenshot-panels.sh DIR OUT_DIR [--only health|findings|page|crawl] ``` Only the first command needs re-running after a panel edit. The script writes `OUT_DIR/panels/*.png` and `OUT_DIR/checks.json`, and exits 1 when a check fails. --- ## Live progress While a crawl runs, `raptor_status` reports `progress` for it, updated about four times a second: - `pages_crawled`, `pages_queued` and `estimated_total`, with the meanings they always had (`null` until the first page is fetched); - `status_counts` (`2xx`, `3xx`, `4xx`, `5xx`), `errors` (fetches that got no answer, which the panel and the digest call failed fetches) and `recent_errors` (the last three, with their `url`, `kind` and `label`); - `current_url`, the page fetched last; - `phase` and `phase_label`, the phase in flight ("Fetching pages", "Reading sitemaps", …) in the same words the CLI uses, and `phases_done` with each finished phase's duration; - `max_pages`, `bar_total` and `bar_label` for a progress bar that never promises more than the page limit ("120 of up to 500 pages"); - `facts_line` and `tally_line`, the two lines the crawl panel prints; - `interrupted`, and `seq`, which counts the writes. A finished job's `result_summary` adds `grade_verdict`, `graded`, `findings_count`, `to_fix`, `duration_ms`, `stopped_at_page_limit`, `urls_not_crawled` and `interrupted`. A failed job's holds only `failure`: what went wrong, in a form the crawl panel can word. --- ## Limits Every digest lists the crawl's limits, the things that change how far you can trust it: a crawl that stopped at its page limit or was interrupted, URLs robots.txt kept it out of, pages that could not be fetched, pages that look built by JavaScript, data saved without its pages, a score from an older scoring model, and a score that stopped at 0. It is the same ordered list the report and the CLI show. The health panel shows the first of them and says how many more there are. When the deductions add up to more than 100, the score stops at 0, and a finding's points then say what it costs rather than how much fixing it alone would lift the score. That explanation sits beside the points wherever they are printed. --- ## Branding The server reads `brand.toml` once at start (from `RAPTOR_BRAND`, or `$RAPTOR_HOME/brand.toml`). If the file is refused, the server still starts, with the Raptor brand, and `raptor_crawl` and `raptor_summary` say so in their text and in `brand.error`, because Claude Desktop hides the server's own log. The panels use the brand's voice and mascot setting, but no logo, accent or agency name. See [report.md](report.md). --- ## Protocol details - Version negotiation on `initialize`; only implemented capabilities are advertised, plus the `io.modelcontextprotocol/ui` extension (see Panels). - Requests are handled **concurrently**, so responses may arrive out of request order — correlate by `id`, as JSON-RPC intends. Each response is one complete newline-delimited line. - `notifications/cancelled` is enforced: in-flight work is aborted and, per spec, no response is sent for the cancelled request. `initialize` is never cancellable. - A tool's result is returned as `structuredContent`, with its written digest as the text block; clients older than `2025-06-18` get the JSON as text instead. - In MCP mode **all logging goes to stderr** — stdout is reserved for JSON-RPC. The log is plain text, with no colour codes, and carries no REST API-key warning, which does not apply to stdio. --- ## REST API | Method | Route | Purpose | |---|---|---| | GET | `/api/v1/health` | Liveness. The only route that never requires auth. | | POST | `/api/v1/jobs` | Create a crawl job | | GET | `/api/v1/jobs` | List jobs | | GET | `/api/v1/jobs/{id}` | Job status and result summary | | DELETE | `/api/v1/jobs/{id}` | Cancel (stopping a running crawl) and delete a job | | GET | `/api/v1/jobs/{id}/results/{filename}` | Download an artifact | | POST/GET | `/api/v1/webhooks` | Register/list webhooks | | DELETE | `/api/v1/webhooks/{id}` | Remove a webhook | Creating a job — note that crawl settings live under `options`, and unknown top-level fields are rejected rather than ignored: ```bash curl -X POST http://127.0.0.1:3001/api/v1/jobs \ -H 'Content-Type: application/json' \ -d '{"type":"crawl","url":"https://example.com","options":{"max_pages":50}}' ``` `crawl` is the only executable job type. The others are refused with a 4xx naming what is supported, rather than accepted and left queued forever. A served `report.html` runs under a locked-down content security policy: no script, no connection and no remote source, with only its embedded `data:` images and fonts allowed. ### Webhooks Registered webhooks fire on terminal job states: `job.queued`, `job.completed`, `job.failed`, `job.cancelled`. Cancelling a running crawl delivers `job.cancelled` immediately, while the crawl stops and saves the pages it fetched, and again when it has. One-shot webhooks remain exactly-once. --- ## Security - **Auth is off unless a key is set.** `--api-key` or `RAPTOR_API_KEY`. Without one, every route except `/health` is open to anything that can reach the port. The default bind is `127.0.0.1`. - **CORS is closed by default.** Allow specific origins with `RAPTOR_CORS_ORIGINS`. - **SSRF protection** on crawl targets and webhook URLs: private, loopback, link-local and cloud-metadata ranges are refused, including IPv4 tunnelled inside IPv6, and the address vetted is the one actually dialled. Override for intranet audits with `RAPTOR_ALLOW_PRIVATE_TARGETS=1`. - Request bodies are capped at 1 MB; artifact downloads are path-traversal safe. --- ## Running more than one process Every Raptor process sharing a `RAPTOR_HOME` shares the job store. Jobs carry an owner and a heartbeat, so an MCP server starting up no longer reclaims a REST server's running crawls. A process can stop only the crawls it runs itself (see Cancelling a crawl). The trade-off: a job orphaned by a crashed process is reclaimed after **120–180 seconds**, not instantly. If you want full isolation — separate job stores per client, say — point each process at its own `RAPTOR_HOME`.