The Web Almanac, an annual report translating the vast HTTP Archive dataset into actionable insights, offers a unique perspective on the state of the web. For the upcoming 2025 report, SEO expert Chris Green, with over 15 years of industry experience and a co-author of the SEO chapter update, shared some surprising discoveries. His analysis reveals little-known facts and unexpected insights, particularly concerning how content management systems (CMS) are shaping technical SEO and the evolving dynamics of bot management and AI crawlers.

“I think the data [in the Web Almanac] helped to show me that there’s still a lot broken. The web is really messy. Really messy.”

Bot Management Is No Longer 'Google, Or Not Google?'

For years, bot management was a relatively straightforward binary choice: allow or disallow Google. However, this landscape is rapidly changing, presenting new challenges for SEOs. Chris Green's research, echoing earlier observations by Eoghan Henn, highlights the increasing complexity, especially with the rise of AI crawlers.

The core issue now revolves around understanding the diverse range of crawlers, their specific intentions, and the precise implications of blocking them. Blocking certain bots can have far greater consequences than others. Furthermore, platform providers must consistently adhere to and appropriately process robots.txt rules, which isn't always the case. The ethical considerations surrounding `robots.txt` and AI crawlers are becoming a critical area for SEOs to master.

Green emphasized that while the Web Almanac report identifies the symptom of `robots.txt` usage, SEOs must proactively grasp how to effectively control these bots. This involves not only understanding each bot's impact but also communicating these nuances to stakeholders within a business. For instance, a team aiming to reduce crawling costs might inadvertently jeopardize AI visibility, while an editorial team might seek to prevent content scraping. SEOs must navigate these internal dynamics, implement technical controls, and articulate the strategic arguments within their organizations. As more platforms and crawlers emerge, SEO teams will need to collaborate across departments to strike the right balance of site access.

“It’s not only understanding what the impact of each [bot/crawler] is, but also how to communicate that with the business. If you’ve got a team who want to cut as much bot crawling as possible because they want to save money, that might desperately impact your AI visibility. Equally, you might have an editorial team that doesn’t want to get all of their work scraped and regurgitated. So, we, as SEOs, need to understand that dynamic, how to control it technically, but how to put that argument forward in the business as well.”

Llms.txt Adoption Surprises Despite Lack of Official Support

One of the most unexpected findings from the report is the adoption rate for the proposed llms.txt standard, which stands at approximately 2% across the dataset's websites. This is particularly surprising given that `llms.txt` has been a contentious topic, with many SEOs questioning its value, and major AI providers have yet to officially commit to its use.

Chris Green admitted that 2% adoption was higher than his expectations. Much of this growth, he believes, is driven by SEO tools that have integrated `llms.txt` as a default or optional feature. However, Green remains skeptical about its long-term impact. Google has repeatedly stated it has no plans to use `llms.txt`, and without clear commitment from dominant AI providers, especially OpenAI, the standard risks remaining a niche, symbolic gesture rather than a functional tool.

Despite this skepticism, Green has observed log-file data indicating that some AI crawlers are indeed fetching these files, and in limited instances, they may even be referenced as sources. He views this less as a competitive advantage and more as a potential parity mechanism, helping certain sites be understood without dramatically elevating their visibility.

“Google has time and again said they don’t plan to use llms.txt which they reiterated in Zurich at Search Central last year. I think, fundamentally, Google doesn’t need it as they do have crawling and rendering nailed. So, I think it hinges on whether OpenAI say they will or won’t use it and I think they have other problems than trying to set up a new standard.”

Different, But Reassuringly The Same Where It Matters

Addressing the challenge of balancing search engine visibility with machine visibility, Green sees “a significant overlap between what SEO was before we started worrying about this and where we are at the start of 2026.”

He acknowledges that while there are distinct systems, different weightings, and varied approaches to interpretation, retrieval, and generation for search engines versus AI machines, the fundamental principles of SEO haven't drastically changed. Green believes that SEO and AI optimization are “kind of the same, reassuringly the same in the places that matter, but you will need to approach it differently” due to the divergence in how outputs are delivered and consumed.

He predicts a shift for SEOs towards greater focus on feeds, feed management, and feed optimization. This evolution is partly driven by initiatives like Google's universal commerce protocol, which could enable direct transactions from search results or AI interfaces, effectively pushing the traditional website out of the direct user journey. However, Green emphasizes, “the information, what we’re actually optimizing still needs to be optimized. It’s just in a different place.”

CMS Platforms Shape The Web More Than SEOs Realize

Perhaps the most significant revelation from Web Almanac 2025 is the profound and often underestimated influence of CMS platforms and tooling providers. Green confessed he hadn't fully grasped the magnitude of this impact, stating, “Platforms like Shopify, Wix, etc. are shaping the actual state of tech SEO probably more profoundly than I think a lot of people truly give it credit for.”

He further elaborated that while individual SEOs are well-intentioned, their overall impact on the web is minimal compared to that of CMS platform providers. Green suggests that for SEOs truly determined to make a broader impact beyond their specific clients, the focus should be on “nudging WordPress or Wix or Shopify or some of the big software providers within those ecosystems.” This dynamic creates a significant opportunity: websites that correctly implement technical standards can achieve substantial differentiation, especially when many sites lag behind best practices.

This insight underscores a broader truth:

“a lot of SEOs believe that Google owes us because we maintain the internet for them. We do the dirty work, but I also don’t think we have as much impact perhaps at an industry level as maybe some like to believe. I think the data in the Web Almanac kind of helped show me that there’s still a lot broken. The web is really messy. Really messy.”

AI Agents Won't Replace SEOs, But They Will Replace Bad Processes

Concluding the conversation, Green addressed the topic of AI agents and automation, clarifying that “Agents are easily misunderstood because we use the term differently.” He stressed that AI agents are not meant to replace human expertise but rather to accelerate processes. Many SEO workflows involve repetitive data gathering and pattern recognition, tasks perfectly suited for automation. The true value of human SEO expertise lies in designing these processes, applying critical judgment, and contextualizing the outputs.

Green estimates that early-stage agents could automate 60-80% of routine SEO work, akin to a highly capable intern. He explained, “It’s going to take your knowledge and your expertise to make that applicable to your given context. And I don’t just mean the context of web marketing or the context of ecommerce. I mean the context of the business that you’re specifically working for.” He argues that many SEOs don't dedicate enough time to customizing their strategies for specific clients. This presents an opportunity to build an 80% automated process, then add significant human value by optimizing the remaining 20% with business-specific logic.

SEOs who embrace agents, refine workflows, and evolve alongside automation are far more likely to remain indispensable than those who resist change. However, Green issued a crucial warning: avoid automating broken processes.

“You need to understand the process that you’re trying to optimize. If the process isn’t very good, you’ve just created a machine to produce mediocrity at scale, which frankly doesn’t help anyone.”

He believes that engaging with AI to improve processes will give SEOs a competitive edge and ensure their longevity as AI becomes more widespread. “I suggest the people that engage with it and make those processes better and show how they can be continually evolved, they’ll be the ones that have greater longevity.”

SEOs Can Succeed By Engaging With The Complexity

The Web Almanac 2025 report doesn't suggest that SEO is becoming obsolete; rather, it highlights an expanding and increasingly complex role that many teams have yet to fully adapt to. While foundational principles like crawlability and technical hygiene remain crucial, they now operate within a sophisticated ecosystem influenced by AI crawlers, content feeds, closed systems, and critical platform-level decisions.

In an environment where technical standards are often poorly implemented at scale, SEOs who deeply understand the underlying systems that shape them can still gain a significant competitive advantage. Automation, when applied correctly, accelerates well-designed processes; conversely, it merely scales inefficiency if processes are flawed. Therefore, SEO professionals who prioritize process design, expert judgment, and business context will continue to be indispensable as automation becomes standard.

In an increasingly “messy” and machine-driven web, the SEOs who thrive will be those willing to engage with this complexity instead of ignoring it. SEO in 2026 isn't about choosing between optimizing for traditional search engines or AI; it's about comprehending how multiple diverse systems consume content and identifying where optimization efforts are most impactful.

Watch the full video interview with Chris Green here:

Thank you to Chris Green for offering his insights and being my guest on IMHO.

More Resources:

Featured Image: Shelley Walsh/Search Engine Journal