LLMs.txt – Why Almost Every AI Crawler Ignores it as of August 2025

Understanding the Decline of LLMs.txt Accessibility: Why AI Crawlers Ignored It as of August 2025

In an insightful analysis conducted by Flavio Longato, an SEO Strategist specializing in Large Language Model (LLM) optimization at Adobe, we gain valuable perspective on the current landscape of AI crawler behavior and the implications for webmasters. The study examines the effectiveness and visibility of the widely discussed LLMs.txt file—a resource intended to communicate site preferences to AI crawlers—highlighting its near-obsolescence among major AI bots as of mid-2025.

Methodology of the Analysis

Longato’s investigation involved meticulously auditing 30 days of raw CDN logs across a substantial sample of 1,000 Adobe Experience Manager domains. The goal was to track which entities made requests to the LLMs.txt file, providing a clear indicator of how different bots interact with it in practice. The findings reveal a stark reality about AI crawler engagement and the evolving behavior patterns across the web.

Key Findings from the LLMs.txt Audit

• LLM-Specific Bots Are No Longer Visiting:
  The most striking observation was the complete absence of requests from prominent LLM-focused crawlers such as GPTBot, ClaudeBot, or PerplexityBot. This suggests these AI entities are either ignoring LLMs.txt or have deprioritized crawling site-specific instructions altogether.

• Google’s Crawler Continues to Lead:
  Google’s desktop crawler remained dominant, accounting for approximately 95% of all detected requests. Its persistent activity underscores its ongoing role as the primary AI crawler, though the relevance of LLMs.txt to Google remains uncertain.

• Bing’s Interest Is Limited and Inconsistent:
  Bing displayed sporadic curiosity, with only seven requests recorded—primarily focused on a single domain. The minimal Engagement hints at a lack of comprehensive crawling policies towards LLMs.txt or perhaps internal prioritization differences.

• OpenAI’s Search Bot Interactions Are Marginal:
  Requests from OpenAIBotSearch were rare, totaling around ten. Notably, GPTBot requests were entirely absent, indicating that OpenAI’s current approach may not involve crawling LLMs.txt or that such efforts have been deprioritized.

• Automated SEO Tools Contributed to Log Noise:
  Various SEO tools, including Semrush Mobile and SiteAudit, generated numerous requests.