Source: Search Engine Journal
Ahrefs' analysis of 137,000 domains reveals that the llms.txt protocol—meant to guide how AI systems crawl and use website content—is almost entirely ignored in practice, with 97% of files receiving zero requests from AI bots. Websites are creating these files to appear responsible, while AI companies' crawlers largely bypass them, leaving the protocol functionally useless as a control mechanism. For publishers and brands worried about content scraping, protection will come through legal leverage, technical barriers, or direct deals with major AI labs—not voluntary machine-readable instructions.