How to Add llms.txt to an Astro Site (Generated, Not Hand-Written)

llms.txt is a proposed convention for giving language models a map of your site: one Markdown file at a well-known URL, listing what exists and where, so a model that lands on your domain can orient itself in a single fetch instead of crawling HTML built for browsers.

This site serves one at /llms.txt, plus the heavyweight companion /llms-full.txt with every article’s full text. Neither is a file I maintain. Both are Astro endpoints that render from the same content collections as the HTML pages, so they update themselves on every build. That’s the version of this idea worth implementing — a hand-written llms.txt is stale by the second post — and the core of it is about sixty lines. This post covers the format, the endpoint code, the details that took iteration (locale splitting, scheduled posts, relative links), and an honest assessment of whether anything actually reads the file yet.

The format, briefly

The llms.txt spec — proposed by Jeremy Howard in September 2024 — is deliberately minimal. It’s Markdown, in a fixed shape:

An H1 with the site or project name (the only required element)
A blockquote summarizing the site in a sentence or two
Optional prose paragraphs with context a model should know
H2 sections containing link lists — - [Title](url): description per line
An optional section literally named ## Optional, which marks links a model can skip when context is tight

The companion convention, llms-full.txt, inlines the full content of everything instead of linking out — one fetch, the whole site in a model’s context window, no crawling at all.

Why Markdown at a fixed URL rather than HTML? Because the audience is a language model at inference time — an agent answering a question right now, budgeting tokens. Your HTML is full of navigation, scripts, and markup overhead; your Markdown is nearly pure signal. The spec’s bet is that sites which hand models clean input get represented more accurately in answers. (Whether anyone’s collecting on that bet yet — see the end of this post.)

Generating it from content collections

A .txt URL in Astro is just a static endpoint: a src/pages/llms.txt.ts file exporting a GET that returns a Response. Everything the file needs — titles, descriptions, dates, tags — is already in the content collection powering the blog, so the endpoint is a map over getCollection:

// src/pages/llms.txt.ts
import type { APIRoute } from 'astro';
import { getCollection } from 'astro:content';
import { SITE_TITLE, SITE_URL, SITE_DESCRIPTION, publishedOnly } from '../consts';

// Folded YAML descriptions contain newlines; the link-list format wants one line.
const oneLine = (text: string) => text.replace(/\s+/g, ' ').trim();

export const GET: APIRoute = async () => {
  const posts = (await getCollection('blog', publishedOnly))
    .sort((a, b) => b.data.pubDate.getTime() - a.data.pubDate.getTime());

  const articles = posts.map(({ id, data }) => {
    const meta = [
      data.category,
      `published ${data.pubDate.toISOString().slice(0, 10)}`,
      ...(data.tags.length ? [data.tags.join('/')] : []),
    ].join(' · ');
    return `- [${data.title}](${SITE_URL}/blog/${id}/): ${oneLine(data.description)} (${meta})`;
  });

  const body = `# ${SITE_TITLE}

> ${SITE_DESCRIPTION}

## Articles

${articles.join('\n')}

## Optional

- [Full article text](${SITE_URL}/llms-full.txt)
- [RSS feed](${SITE_URL}/rss.xml)
- [Sitemap](${SITE_URL}/sitemap-index.xml)
`;

  return new Response(body, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
};

Prerendered like every other page, deployed as a plain file, zero runtime cost. Three details in there that are easy to get wrong:

Absolute URLs. Root-relative links are fine in HTML that’s read in place; llms.txt is read out of place, pasted into a context window far from your origin. Every link gets the full https:// form.

Filter unpublished posts. The endpoint must use the same draft/schedule filter as the rest of the site — that’s the publishedOnly in the collection call. This site does scheduled publishing on a static build, and an early version of this endpoint skipped the filter, which would have cheerfully announced future posts to any model that looked. Anywhere content is listed, the filter goes too.

Metadata in the description line. The spec only asks for title-plus-description, but dates, category, and tags cost a few tokens and give a model exactly what it needs to answer “is this current?” or “what does this site cover?” without fetching anything else.

llms-full.txt: the whole site in one file

The full-text variant is the same shape, but instead of linking to each post it embeds the post’s Markdown source. Collections hand you that directly — post.body is the raw Markdown with frontmatter already stripped:

const articles = posts.map((post) => {
  const { title, description, pubDate } = post.data;
  const header = [
    `URL: ${SITE_URL}/blog/${post.id}/`,
    `Published: ${pubDate.toISOString().slice(0, 10)}`,
  ].join('\n');
  return `# ${title}\n\n${header}\n\n> ${oneLine(description)}\n\n${absolutize(post.body ?? '')}`;
});

const body = `# ${SITE_TITLE} — full article text\n\n${articles.join('\n\n---\n\n')}\n`;

One transformation matters here: internal links. Posts link to each other root-relatively ([deploy guide](/blog/...)), and in an extracted blob those links point nowhere. A one-line rewrite fixes every one:

const absolutize = (markdown: string) =>
  markdown.replace(/\]\(\/(?!\/)/g, `](${SITE_URL}/`);

The negative lookahead leaves protocol-relative //example.com URLs alone.

Two caveats before you ship one of these. If your posts are MDX with heavy component usage, post.body is the source — JSX tags and all — which may or may not be what you want a model to read; this blog’s posts are almost pure Markdown, so the source is clean. And the file grows linearly with your archive (this site’s is a couple thousand words per article across a dozen articles — still tiny by web standards, but a 500-post archive should probably offer per-page .md instead).

The multilingual wrinkle

This blog publishes in English and Japanese, which raised a question the spec doesn’t address: one bilingual file, or one per language? I went with per-locale files — /llms.txt and /ja/llms.txt — on the logic that a model that arrived on a Japanese page should get Japanese titles and Japanese URLs, not a blob interleaving two languages. Each file lists its sibling under ## Optional, so the other edition is discoverable without being mixed in. In Astro this falls out naturally: the endpoint moves into the [...locale] routing directory and getStaticPaths emits one file per language.

Does anything actually read it?

The honest section. As of mid-2026: no major crawler has committed to consuming llms.txt, and Google’s search folks have been openly dismissive — John Mueller compared it to the keywords meta tag. Nobody serious claims a measurable citation lift, and you should be suspicious of anyone selling one.

What is true: adoption on the publishing side is real (Anthropic’s docs serve one, and docs platforms generate them by default), AI crawler traffic itself is very real — I’ve measured what AI agents do against this site — and agents that fetch pages on demand can use the file today even though bulk crawlers ignore it, because it’s just Markdown at a guessable URL. Anecdotally, that’s where I’ve seen it consulted: agentic fetchers, not index crawlers.

So the case is not “this will boost your AI visibility” — nobody can promise that. The case is that the generated version costs a screenful of code once and zero maintenance forever, the payoff if the convention lands is real, and unlike most AI-SEO advice it cannot hurt: it’s an additive plain-text file that no human ever sees. Cheap lottery tickets are worth holding when they renew themselves on every build.

Checklist

src/pages/llms.txt.ts — H1, blockquote, link sections; generated from getCollection
src/pages/llms-full.txt.ts — full post.body per article, internal links absolutized
Same published/draft filter as the HTML pages
Absolute URLs everywhere; Content-Type: text/plain; charset=utf-8
Multilingual sites: one file per locale, cross-linked under ## Optional

Then curl https://your-site/llms.txt | head after deploying, and forget about it — the next build keeps it current, which was the whole point of generating it.

The format, briefly#

Generating it from content collections#

llms-full.txt: the whole site in one file#

The multilingual wrinkle#

Does anything actually read it?#

Checklist#