A slow, polite reader the owner asked for.
The API recovery scan reads a site's API errors and sitemap pages and says where people and agents hit a dead end. Only a team that has proved it owns the domain can start one, and it confirms each time that it may send the traffic.
How to recognise it
- User agent:
site-monitor-scan/1.0 (+https://sitesinthewoods.com/scanner) - Header:
X-Scan-Id, the scan's id: tell us it if you want us to look at one. - Requests: GET only, on the site's own host and port, no cookies, no credentials, one at a time, at least half a second apart (the owner may choose slower). It reads the sitemap's pages and asks the API's paths without a key; it never follows links it finds on pages (from an API error it may open the one docs page the error points to, on the same host), never sends a form, and skips anything that looks like an action (log out, delete, export, unsubscribe).
How to slow or stop it
- A
User-agent: site-monitor-scangroup in robots.txt is always obeyed, for pages and API requests alike:Disallow: /stops it before it starts. (The one-look check on /check is a different visitor,site-monitor: to keep both out, name both.) robots.txt is read again whenever a scan starts or resumes, and one that answers an error or nothing stops the scan. Crawl-delayis honoured, up to 10 seconds between requests.- A run of 429 or 503 answers, 403s to its page requests, or a bot challenge, pauses it for five minutes; three pauses end the scan. We never work around a block.
- Anything else: [email protected], and we stop it.
Our monitor
The checks themselves (uptime, TLS, sitemaps, agent files, MCP servers) come from a different visitor: site-monitor/1.0, linking to this page. It's there because a team added the site to watch: the homepage about once a minute, sitemaps and agent files every half hour to hourly, plus the homepage and llms.txt under a few AI crawlers' user agents, to see what they get. It sends no cookies or credentials, refuses private addresses, and doesn't read robots.txt for its uptime checks (it isn't crawling). If it's turning up where it shouldn't, tell us and we'll stop it. The same checks are open source as agent-ready-uptime, which names itself agent-ready-uptime.
What the fixes mean
- Errors that say how to recover
- A 404 or 401 whose body names what to try next: a
detail(RFC 9457), asuggestion,next_steps, or a sentence with a link. - Docs one hop away
- An error that links to documentation on the same site, which answers. Agents recover through the docs as much as through the error.
- Paging
- A list that says how to get the next page (
Link: rel="next", anextURL, a cursor) and when to stop (has_more, a total). - The sitemap
- Every URL it lists should answer.
- The not-found page
- Answers 404 and links back into the site.
- Pages nothing links to
- Reachable only from the sitemap.
- Deep pages
- More than four clicks from the homepage.
- A way back
- A child page links to its parent (a breadcrumb).
- Every link on the page is in its header, nav or footer.
- A way on
- The page's content links somewhere next.
- Clear labels
- Links say where they go, not "click here" (WCAG 2.4.4).
- Too many next steps
- More than five, or very long ones.
- Links in answers
- Successful API answers that link to related resources. The evidence that agents need these is thin.
Last updated .