TripleVisibleBot
TripleVisibleBot is the crawler of TripleVisible. A site's owner adds the site to TripleVisible to measure how well it is found by search engines, answer engines and AI agents; this crawler reads the site's public pages for that owner. It is not a search index.
How it identifies itself
Every request carries these headers:
User-Agent: TripleVisibleBot/1.0 (+https://triplevisible.com/bot) Signature-Agent: https://triplevisible.com Signature-Input: ... Signature: ...
We never use another crawler's name and never present ourselves as a browser.
What it fetches, and how often
- It reads
/robots.txtfirst and obeys it on every request, including after a redirect. If robots.txt cannot be read because of a server error, it fetches nothing from that site until it can. - It only reads:
GETandHEADrequests for public pages, sitemaps and files. It never submits a form, never logs in, never sends data to a site and never follows a redirect off the site it was asked to read. - Rate: one request per second at most today. It never exceeds two requests at once, and never exceeds four per second even where a site's owner has turned on a faster mode.
- It stops when a site's bot protection challenges it. It does not retry a challenge, and it does not try to get around one.
- A
429response halves its rate for the rest of that run, and aRetry-Afteris honoured for up to 30 seconds. - For a small sample of pages it may load the page in a real browser, with the same identity, to see what a visitor sees.
How to verify a request is ours
Requests are signed as HTTP Message Signatures (RFC 9421, the Web Bot Auth profile) with an Ed25519 key. The signature covers the request's authority and the Signature-Agent header, and names the key by its JWK thumbprint (keyid). Our public keys are published as a JWK set at:
https://triplevisible.com/.well-known/http-message-signatures-directory
When we rotate a key, the directory lists the old and the new key together for at least 30 days. A request that names neither the user agent, the Signature-Agent header nor a key in the directory is not from us; please tell us.
How to block or slow us
Use robots.txt. We match the product token TripleVisibleBot (any letter case) and use the group that names it; if none does, the User-agent: * group.
User-agent: TripleVisibleBot Disallow: /
That blocks us from the whole site. To block part of a site, list the paths you want left alone. To slow us down:
User-agent: TripleVisibleBot Crawl-delay: 10
We honour Crawl-delay: the gap between the starts of our requests is the larger of one second and your value. It is not capped by us, so a large value means fewer pages in a run; a run ends at its time limit and reports what it did not read. We understand Allow and Disallow with * and $, and the longest matching rule wins. Only the rules of robots.txt govern our crawler; a Content-Signal line is not read as an instruction to it.
If you would rather we did not read your site at all, or a bot-protection service in front of it blocks us, you can also block the user agent or the Signature-Agent value above.
Contact
Questions, complaints and requests to stop: [email protected]. Please include the host that was read and the time of a request.