NEW Feature: Visit your customized dashboard and sync all your results on the cloud.
Analyze 1 page
Get an instant LLM visibility score and know exactly what to fix.
Compare 2 pages
Analyze two pages simultaneously and see exactly where you win, lose, or tie across every pillar.
Website Audit
Crawl your entire sitemap and get an aggregate LLM visibility score for every page.
Scan History
Track your scores over time and measure the impact of your optimizations.
What questions people ask
Discover real questions people are asking about any topic, powered by AI.
AI Crawler Tester
Test AI crawlers against your pages and see who is blocked and who gets through.
llms.txt Generator
Generate a compliant llms.txt file that helps AI systems understand your site at a glance.
robots.txt Generator
Control which AI crawlers can access your content with a properly configured robots.txt.
JSON-LD Schema Generator
Build structured data markup that gives LLMs explicit signals about your content's meaning.
Meta Tags Generator
Generate complete, optimized meta tags including Open Graph and canonical for any page.
MCP Connector
Connect Claude and other AI agents to the hey-eye analyzer and run visibility checks straight from the chat.
Blog
AI visibility insights, optimization strategies, and practical guides for LLM-ready content.
Support
Guides, tutorials, and documentation for hey-eye.
Structural Integrity 30%
Title, headings, canonical, Open Graph, semantic HTML and language attributes.
AI Extractability 35%
JSON-LD schema, paragraph length, lists, date signals, internal links and breadcrumbs.
Content Clarity 15%
Sentence length, heading density and Flesch readability — with Greek language support.
Authority & Trust 20%
Author attribution, social profiles, image alt text, meta robots and robots.txt AI access.
Analyze 1 page Compare 2 pages Website Audit Scan History
Questions Asked AI Crawler Tester llms.txt Generator robots.txt Generator JSON-LD Schema Generator Meta Tags Generator
MCP Connector Blog Support Structural Integrity AI Extractability Content Clarity Authority & Trust
Login Dashboard Contact

What is robots.txt?

robots.txt is a plain-text file at a site's root that tells crawlers which paths they may request. It is a directive, not an enforcement mechanism.

Published

Definition

robots.txt is a plain-text file at the root of a domain that tells automated clients which paths they may request. It dates from 1994 and was standardized as RFC 9309 in 2022.

It is a set of instructions that well-behaved crawlers choose to follow. It blocks nothing by itself.

Why It Matters for AI Visibility

It is the first gate, and the one most often closed by accident. A single inherited Disallow rule, or a wildcard written to stop scrapers, can remove a site from AI search entirely — and nothing in your analytics will report it.

The second trap is subtler. Because robots.txt is only a request, a site can allow a crawler there while a firewall, CDN or bot-management service blocks the same crawler at the network layer. The two live in different systems and disagree more often than people expect. Checking robots.txt alone tells you what you intended; only a live fetch with the bot’s real user-agent tells you what actually happens.

Example

User-agent: GPTBot
Disallow: /private/
Allow: /

Sitemap: https://example.com/sitemap.xml