Skip to content

Tools

Can AI read your page?

AI tools send crawlers, small programs that read pages, before they quote one. This tool shows which crawlers your site lets in and what they find when they arrive.

  1. Type a page address. Add facts that must be on the page, if you like.
  2. The tool reads your robots.txt and that one page, the way a crawler does.
  3. Read the result and work down the Fix first list.

Free. The tool reads two public files, robots.txt and the page, and stores nothing. It introduces itself as AiLatelyCrawlerCheck and follows the site's own robots.txt. Updated November 10, 2026.

How it checks. It matches robots.txt the way the internet standard (RFC 9309) and Google's open-source parser do: the rule that names a crawler wins over the * rule, the longest match wins, and Allow wins a tie. It checks 9 crawlers that answer questions and 7 that train models. Names and jobs follow the ai-robots-txt project's public list, read on November 10, 2026.

What it cannot see. A crawler can ignore robots.txt, and some do. A page can also block a crawler at the server with a rule robots.txt never shows. The tool reports what the files say. For how the page reads to a person and to an AI, run the Search Checklist.

Keyboard

j k
Move through a list
Enter
Open the selected piece
/
Search, and browse the columns
g then r
Right Now
g then a
Articles
g then s
The Signal
g then o
Opinion
g then b
Analysis
g then p
People
?
This sheet
Esc
Close

AI Right Now is free with an account.The Citation Audit is free with an account.Skills are free with an account.Tutorials are free with an account.You have read your 3 free articles.

Every article and comic, AI Right Now, the Citation Audit, Skills and Tutorials.

Join Ai Lately for free

Join with Google or your email. Already a member? Sign in

Press anywhere else for the home page.