Key Takeaways:
- llms.txt does not affect Google rankings. Google has confirmed, in its own Search Central documentation, that Google Search does not use llms.txt for ranking, AI Overviews, or AI Mode, and never has.
- It still helps in one narrow case, and we recommend adding it anyway. llms.txt genuinely helps AI coding agents like Cursor, Claude Code, and Windsurf read developer documentation cleanly. It’s not a ranking lever, but it’s inexpensive to implement, carries virtually no downside, and means you’re prepared if support for the format becomes more widespread.
- Google is building separate tooling for AI agents. Lighthouse and PageSpeed Insights now include an experimental “Agentic Browsing” category that checks llms.txt, accessibility, and layout stability. This is a different concern from Search rankings.
- The real story is the direction, not the file. AI agents browsing and acting on websites are becoming a measurable part of the web, and Google’s tooling is starting to reflect that, even while its ranking systems stay unmoved.
- OOm helps clients separate hype from substance. Through SEOCloud, our in-house SEO/GEO Analysis platform, we track what’s actually proven to move visibility in AI search, instead of chasing whichever file format is trending.
Introduction
llms.txt has been one of the most argued-about files in SEO over the past year. Some experts swear by it. Google’s own Search team has called it pointless, comparing it to the discredited meta keywords tag. And now Google’s own tools are quietly checking for it.
This article breaks down what llms.txt actually is, why the disagreement exists, and what Google’s new “Agentic Browsing” feature inside PageSpeed Insights tells us about where things are heading.
In June 2026, Google updated its Search Central documentation to say plainly that Google Search does not use llms.txt files at all. Not for ranking. Not for AI Overviews. Not for AI Mode.
That’s not quite the whole story, though. Around the same time, Google’s Chrome team added a new “Agentic Browsing” category to Lighthouse and PageSpeed Insights, the same free tool millions of marketers use to check page speed. One of the checks in that category is whether your site has an llms.txt file.
So does Google care about this file or not?
It depends on which part of Google you ask, and what you’re actually trying to achieve.
What Is llms.txt?
llms.txt is a plain-text Markdown file placed at the root of a domain (e.g., yoursite.com/llms.txt). It acts as a curated index of a website’s most important pages, stripped of navigation menus, ads, and JavaScript clutter, so an AI model can quickly understand what a site is about.
The format is deliberately minimal: an H1 with the site name, a short blockquote summary, and H2 sections linking out to key pages with one-line descriptions. There’s also an optional companion file, llms-full.txt, which compiles a site’s documentation into one Markdown file.
It’s often compared to two files webmasters already know. robots.txt tells bots where they can’t go. sitemap.xml lists everything that exists. llms.txt is meant to highlight what matters most.
Who created it, and why
llms.txt was proposed on 3 September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai. The problem he set out to solve was narrow: LLM context windows are too small to absorb an entire website, and standard HTML pages are full of navigation chrome, scripts, and markup that waste tokens an AI agent could spend on actual content.
His own project, FastHTML, became the reference implementation. The intended audience was specific: developers and AI coding agents that need fast, clean access to API documentation, not general web search visibility. Adoption stayed niche until November 2024, when Mintlify rolled out automatic llms.txt generation for every site it hosted. Within weeks, thousands of developer docs sites, including Anthropic, Cursor, Stripe, and Zapier, had one.
Why Some Experts Recommend It
Once credible names like Anthropic, Stripe, and Cloudflare started shipping llms.txt files, the logic in marketing circles became simple: if these companies are doing it, surely it influences how AI tools see a site. That assumption fed directly into the emerging disciplines of GEO (Generative Engine Optimisation) and AEO (Answer Engine Optimisation), the practice of trying to get cited inside ChatGPT, Perplexity, and AI Overviews.
The timing mattered too. AI Overviews and AI Mode were expanding fast, and marketing teams under pressure to “do something about AI visibility” had few concrete levers to pull. A small Markdown file felt like an easy, low-cost win compared to the vague advice that otherwise dominates GEO discourse.
There’s a real technical kernel underneath the hype, though. llms.txt genuinely helps AI coding agents like Cursor, Claude Code, and Windsurf pull clean documentation instead of guessing from cluttered HTML. The confusion came from stretching that narrow benefit into a broad SEO ranking claim it was never designed for.
Where the data falls apart
AI coding tools do meaningfully use llms.txt when developers point them at a documentation site, and even critics of the file agree this is real. General-purpose AI search and chat tools tell a different story. One 90-day study of AI bot traffic found that out of 62,100 AI bot requests to a monitored set of sites, only 84 ever touched the site’s llms.txt, about 0.1%, performing worse than an average forgotten page on the same domain. Another firm that monitored more than 500 million AI bot traffic events found that GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip the file and crawl HTML directly.
GEO consultant Kai Spriestersbach has been among the most direct critics, arguing that llms.txt “has not been used by any major AI search system in a retrieval context after more than a year,” and that Jeremy Howard “never proposed llms.txt as a GEO or SEO measure” in the first place.
Google’s Stance: Why It “Isn’t a Thing” for Search
In June 2026, Google updated its AI Search optimisation guide under a section titled “Clarifying guidance on llms.txt files,” saying it added the note to address questions from the community. The key line, bolded by Google for emphasis: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.”
And on whether to remove an existing file: “It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files… Doing so won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them.”
This isn’t a policy change. Google has never used llms.txt as a ranking signal, so nothing actually flipped. What changed is that Google felt the need to say so explicitly, which tells you how widespread the misconception had become.
It wasn’t the first time Google had to address it. In December 2025, an llms.txt file briefly appeared on Google’s own Search Central developer documentation and several other Google properties. SEO commentator Lidia Infante spotted it. Google’s John Mueller responded on Bluesky with a cryptic “hmmn :-/” before the file was pulled within hours. The likely explanation was an internal CMS update, not a deliberate Search team decision. When asked directly whether the file’s presence was an endorsement, Mueller said simply, “no.” On a separate occasion, he compared llms.txt to the long-dead meta keywords tag and called the idea of building separate Markdown pages for bots “a stupid idea.” Google’s Gary Illyes confirmed at Search Central Live Deep Dive Asia Pacific that Google does not support llms.txt and has no plans to.
Google’s New Agentic Browsing Feature in PageSpeed Insights
What it is
In May 2026, Google added a new audit category to Lighthouse called Agentic Browsing, sitting alongside the familiar Performance, Accessibility, Best Practices, and SEO categories. Because PageSpeed Insights runs on Lighthouse, the new category flowed through to pagespeed.web.dev within weeks, so it now silently appears on reports run by people who have no idea it was added.
What it measures
Unlike other Lighthouse categories, Agentic Browsing doesn’t produce a single 0 to 100 score. It reports a pass ratio (e.g., “3/3”) across up to four checks: whether an llms.txt file exists and follows the spec, whether the site supports WebMCP (an experimental protocol that lets agents call structured “tool contracts” instead of guessing through the interface), whether the accessibility tree is properly named and structured, and whether the page’s layout stays stable (CLS).
Only the first three currently count toward the default pass ratio. WebMCP stays excluded from the headline score while the spec settles.
Why Google introduced it
A growing share of requests hitting public web servers now come from AI agents rather than humans: tools like OpenAI’s Operator, Anthropic’s Computer Use, Google’s own Project Mariner, and ChatGPT’s browsing mode. These agents get a task, such as comparing products and booking the cheapest one, and try to execute it across multiple sites with no human steering the mouse. Most websites, built for a human eye and a graphical browser, aren’t structured for that.
Google is effectively asking a parallel question to traditional SEO: not “can a search crawler index this page,” but “can an AI agent read this page and act on it.” That covers things like filling out a checkout form correctly or recognising that an “Add to Cart” button is something it can click.
Lighthouse has a track record of turning experimental categories into industry standard practice, the way it did with Core Web Vitals and Accessibility. Agentic Browsing followed that path within weeks, moving straight into the default PageSpeed Insights configuration that now runs automatically on millions of reports.
So is Google contradicting itself?
Not exactly. Two different Google teams are answering two different questions. Google Search asks: does this file affect how pages get discovered, crawled, or ranked? Its answer is a clear no. Google Chrome / Lighthouse asks: once an agent has found this page, is it built well enough for a machine to use? Its answer, increasingly, is that llms.txt is one small but useful signal.
John Mueller put it this way when pressed on the apparent irony: it’s worth separating “discovery” (finding a website through a search engine) from “functionality” (once someone has found the page, helping them do the task they want). Search is a discovery problem. Agentic Browsing is a functionality problem. llms.txt was never going to solve the first, and was only ever meant to help with the second.
SEOCloud: Measuring What Actually Matters
For businesses in Singapore, the real challenge isn’t just understanding what Google says about llms.txt. It’s knowing whether your brand is actually showing up where customers are searching now, inside AI Overviews and AI-generated answers.
This is exactly what our in-house platform, SEOCloud, was built to track. It’s GEO Analysis module breaks down which of your target keywords appear in AI Overview results and which don’t, compares clicks and impressions between AI Overview and non-AI Overview keywords over time, and benchmarks your domain’s AI Overview visibility against competitors. Instead of guessing whether a tactic like llms.txt is doing anything for your AI visibility, this gives our team an actual, measurable answer.
That’s the same standard we’d apply to any new AI SEO tactic: don’t take it on faith because it’s trending, check whether it moves anything for your specific site.
What This Means for SEO and GEO
Strip away the file format and a clearer trend shows up. Google isn’t ignoring the rise of AI agents browsing the web on people’s behalf. It’s refusing to let that trend leak into its ranking systems through an unverified, self-declared file that any site could stuff with misleading claims, the same failure mode that killed the meta keywords tag.
Instead, Google is building separate tooling for the “agent readiness” problem, tooling that rewards things SEOs already know how to fix: clean semantic HTML, a coherent accessibility tree, stable layouts. The genuinely new piece is WebMCP, and it isn’t ready for serious attention yet.
This lines up with Addy Osmani, Director of Engineering at Google Cloud AI, who argues that AI agents represent a third audience for content, alongside human readers and search crawlers. Even Osmani flags in his own writing that the Google Search team “does not officially recommend llms.txt as a standard,” even as Chrome colleagues build tooling that scores it. Longtime Google watcher Glenn Gabe put it well: “I think at some point, Google needs to come out with an official stance on all of this. Because as of now, the various teams are not aligned really.”
For businesses, the takeaway is that AI search readiness and “AI agent readiness” are related but not the same discipline, and no single file satisfies both. The companies best placed over the next year will be the ones building real foundations now, not the ones chasing whichever acronym is trending this month.
Practical Tactics for Businesses Today
Here’s how we’re actually advising clients on this right now, not in theory.
a) Don’t expect llms.txt to move your Google Search rankings, but it’s worth adding anyway
The file does not help or hurt visibility, clicks, or rankings in Google Search, AI Overviews, or AI Mode. Google has confirmed that directly, and today’s AI search and chat crawlers overwhelmingly ignore it as well. Even so, we still recommend that most clients add a basic, well-formed llms.txt. It takes only a few minutes to implement, there’s very little downside, and you’ll already be covered if support for the format becomes more widespread. It also makes life easier for developers using AI coding tools like Cursor or Claude Code with your API or product documentation. Think of it as a low-cost precaution, not the centrepiece of your AI SEO strategy.
b) Run your key pages through PageSpeed Insights
Check the new Agentic Browsing category. It’s free, it runs automatically, and it surfaces useful diagnostics on accessibility and layout stability that matter for human users and AI agents alike.
c) Prioritise fundamentals over format chasing
Useful, accurate, well-structured content on crawlable HTML pages remains the foundation for both SEO and GEO. Watch experimental pieces like WebMCP, but don’t rebuild your site around a moving target.
FAQs
Does llms.txt help with Google rankings?
No. Google has confirmed directly in its Search Central documentation that Google Search does not use llms.txt for ranking, AI Overviews, or AI Mode. Having one won’t help your visibility, and removing one won’t hurt it either.
Should I remove my llms.txt if I already have one?
Not necessarily. If your documentation is used with AI coding tools such as Cursor or Claude Code, the file can still provide value. Just don’t expect it to improve your Google Search rankings or AI visibility.
Do ChatGPT and Perplexity actually read llms.txt?
Rarely. Independent log-file studies suggest that AI search and chat crawlers overwhelmingly skip the file and crawl HTML pages directly. The clearest exception is AI coding tools reading developer documentation.
What is Google’s Agentic Browsing feature in PageSpeed Insights?
It’s a new, experimental Lighthouse category that scores how ready a page is for AI agents to read and act on, covering llms.txt, accessibility structure, layout stability, and an emerging protocol called WebMCP. It has nothing to do with Search rankings.
Should my business worry about AI agents browsing our website?
Yes, but there’s no need to panic. Clean HTML, a coherent accessibility tree, and stable page layouts help both AI agents and human visitors, and most of that work overlaps with good SEO practice you should already be doing.
Conclusion
The llms.txt story is a useful lesson in how AI search hype spreads. A narrow technical fix for AI coding agents got reframed by the wider SEO industry as a universal AI visibility tool, and Google eventually had to step in and say so, plainly, in its own documentation. At the same time, Google’s Chrome team is building genuinely new tooling for a real trend: AI agents that browse, read, and act on websites on a user’s behalf.
The lesson for businesses isn’t to ignore AI agents. It’s to build for clean structure, accessibility, and stable layouts because both AI agents and human users benefit from them, not because a single Markdown file will move your rankings.
Take the lead now. Request a complimentary AI Visibility and Technical SEO Review and speak with our experts to find out what’s worth doing for your site, what can wait, and what isn’t worth your budget.
About OOm
OOm has been a leading digital marketing agency in Singapore since 2006, with expertise in Search Engine Optimisation (SEO), Generative Engine Optimisation (GEO), Search Engine Marketing (SEM), social media marketing, and website design and development. Our clientele ranges from small and medium businesses to enterprises.
OOm has won over 30 marketing awards over the years, including Agency of the Year, Lead Generation Agency of the Year, Independent Agency of the Year, E-Commerce Marketing Agency of the Year, and Excellence in Search Marketing. OOm is also a Google Premier Partner, reserved for the top 3% of agency partners in Singapore, and an official Meta Business Partner.
Sources and Further Reading
- Google Search Central, AI optimisation guide: “Clarifying guidance on llms.txt files” (June 2026)
- Google Chrome for Developers, Lighthouse agentic browsing scoring
- Search Engine Land, coverage of Google’s llms.txt guidance and the Agentic Browsing rollout
- Search Engine Roundtable, coverage of Google’s statements on llms.txt
- Addy Osmani, “Agentic Engine Optimization (AEO),” addyosmani.com (April 2026)
- Kai Spriestersbach, “The llms.txt is dead. More precisely: a dud,” Medium (February 2026)
- llmstxt.org, original specification by Jeremy Howard, Answer.AI (3 September 2024)
