robots.txt is a sign, not a fence: 8 technical vectors through which AI still reads your website
Dev.to AI
•
Generative AI
Robotics
AI Research
You configure robots.txt like this: User - agent: GPTBot Disallow: / User - agent: CCBot Disallow: / User - agent: anthropic - ai Disallow: / User - agent: PerplexityBot Disallow: / User - agent: * Disallow: / You enable Cloudflare Bot Management. You set up Akamai. Maybe even a server-side paywall. And then you query ChatGPT about your product and it cites your website as a source. How? I work on GEO (Generative Engine Optimization) projects where we audit how LLMs represent brands. We routinely analyze thousands of prompt-response pairs.