What is Content-Signal in robots.txt?
Content-Signal is a robots.txt directive that states how your content may be used after it is crawled, separately for search, AI answers and AI training.
Published
Definition
Content-Signal is a robots.txt directive, introduced by Cloudflare in 2025, that states how a site’s content may be used after it has been crawled — declared separately for search indexing, AI answer generation, and AI model training.
robots.txt answers “may you fetch this?”. Content-Signal answers “and what may you do with it once you have?”.
Why It Matters for AI Visibility
Until now the choice was binary and blunt: allow a crawler or block it. But most publishers do not want the same answer for all three uses. Appearing as a cited source in an AI answer is valuable; being absorbed into a training set may not be. A single Disallow cannot express that difference, so many sites blocked everything and lost the visibility along with the training.
Content-Signal is a stated preference, not an enforcement mechanism. It carries weight because it is machine-readable and explicit, not because it is technically binding.
Example
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /This says: index me for search, use me to ground AI answers, do not train on me.
hey-eye.gr publishes its own signal with all three set to yes.
Related
- robots.txt · llms.txt · GPTBot
- Tool: robots.txt Generator