Lost LLM

Using (generative) AI systems, particularly chatbots such as ChatGPT, Gemini, or Perplexity, for search is becoming more and more common. While the technical details of these systems, including, for example, interesting RAG questions, are fascinating, it all comes down to LLMs processing website information.

That said, most websites are built for humans, and while LLMs can process the content – usually after another piece of software has extracted it – it is not optimized with LLMs in mind. The introduction of a /llms.txt file, akin to, e.g., /robots.txt tries to solve this issue (see proposal).

Put simply, llms.txt is a markdown file that is optimized to be read and processed by LLMs, allowing for better access to the data on the website and also, possibly, as an LLMO/GEO – the generative AI analog to SEO – measure.

As I find the idea interesting, I’ve implemented a basic version of this for this website. You can find it here llms.txt. It contains essential information about this website, as well as links to key pages and articles, each accompanied by a brief description. Furthermore, these key pieces of content are also made available in markdown to make them as accessible to LLMs as possible.

From a technical point of view, I am relying on a modified version of Kyle Gao’s jekyll-llmstxt plugin. So far it works quite well, and I really enjoyed how easy it was to modify the plugin to my need.

However, this simplicity leads me to my final point: While llms-txt is a great idea, there is a lot to be learned about how to implement it best. Of course, providing LLMs with LLM-optimized content in markdown will help, for example, in RAG use cases. However, the key component will still be the scraping and handling of the data and continuously learning how different systems will handle the file and its content.

In any case, I am interested in whether I will be able to see any changes in my analytics data. I will also use this as an opportunity to play with the llms.txt and different approaches to structuring it. So far, at least o3 told me that it likes my file. I will gladly take this, without thinking about it too much, as a compliment and as a good starting point for further experimentation.