Source: Search Engine Journal
Major publishers and brands that aggressively block AI crawlers via robots.txt are now paying search platforms and AI companies for visibility they previously owned organically. The sequence is direct: block training data access to protect IP, lose algorithmic ranking signals, then purchase ads and partnerships to compensate for lost discoverability. This creates a revenue arbitrage for platforms like Google, OpenAI, and Perplexity, who extract payment from both sides of the content supply chain while brands absorb the cost of their own protection strategy.