How we score a site.
The score is what a site earns over what it could: a check that passes earns its weight, one that only partly passes earns half, and a missing one nothing. Checks that don't apply to a site, and checks we couldn't run this time, don't count either way. Newer standards count at half weight.
The checks
| Check | Standard | Tier | Weight |
|---|---|---|---|
robots.txt robots | RFC 9309 | established | 1 |
Googlebot may read the homepage and llms.txt robots.googlebot | RFC 9309 | established | 2 |
Bingbot may read the homepage and llms.txt robots.bingbot | RFC 9309 | established | 2 |
OAI-SearchBot may read the homepage and llms.txt robots.oai_searchbot | RFC 9309 | established | 2 |
Claude-SearchBot may read the homepage and llms.txt robots.claude_searchbot | RFC 9309 | established | 2 |
PerplexityBot may read the homepage and llms.txt robots.perplexitybot | RFC 9309 | established | 2 |
The homepage may be indexed crawlers.noindex | robots meta tag | established | 3 |
A description share.description | HTML meta description | established | 3 |
A share card title share.og_title | Open Graph | established | 1 |
A share card image share.og_image | Open Graph | established | 2 |
Structured data that parses share.structured_data | schema.org JSON-LD | established | 2 |
llms.txt agents.llms_txt | llmstxt.org | common | 3 |
security.txt agents.security_txt | RFC 9116 | established | 2 |
An OpenAPI description (sites with an API) agents.openapi | OpenAPI | common | 3 |
An MCP manifest (sites with an API or MCP server) agents.mcp_manifest | MCP | common | 3 |
An agent card (sites with an agent) agents.agent_card | A2A | emerging | 1 |
A site that publishes an OpenAPI description, an MCP manifest or an agent card is scored as a site with an API: then a missing OpenAPI description or MCP manifest counts against it. Otherwise they don't count.
Grades
- A · Ready: 90 and up
- B · Mostly ready: 75 and up
- C · Needs work: 60 and up
- D · Hard to use: 40 and up
- F · Closed: below 40
A homepage that asks not to be indexed, or a robots.txt that stops every search and AI crawler we check, holds the grade at D (at most 59), whatever else passes.
Why scores differ between tools
Other scanners check other things, weigh them differently and decide differently what applies: on the same day, two tools running the same engine gave one site 88 and 96. Ours is a look at the homepage and its public files, not a security test or a certification. The weights are also data.
Changes
- ·
c2A score and grade from the weights below, four questions, an app or API scope, and a cap at D for a site that keeps search engines out. Optional files a site doesn't need no longer count; checks we couldn't run don't either. - ·
c1Reports kept at each site's own address, with a stable id per finding.
Last updated .