Optimising a Blog to Be the Source AI Assistants Cite

Search traffic to this blog has been flat for a couple of years and I’ve stopped expecting it to move. So I went looking at where visitors come from instead, and whether the AI assistants are among them. They are, barely. In the twelve months to 16 September 2026, Analytics counted 5,372 sessions here. Twenty-seven of them came from an AI assistant: twelve from chatgpt.com, nine from gemini.google.com, six from Perplexity, and none from claude.ai. Half a percent. The ChatGPT visitors were the good kind, though. They stayed five minutes on average, against 27 seconds for direct traffic and three minutes for Google, because they’d already read the answer and wanted the detail.

So I’ve spent some time on how a small blog gets picked as the source for those answers. This post is what I’ve done on this site and what the logs say about it. Where I have numbers I give them, and where I’m guessing I say so. Three sentences in the first draft turned out to be wrong once I looked, and I’ve left the corrections in.

Two kinds of bot, and only one of them cites you

#

The first thing to get straight is that “AI bots” is two different jobs with different user agents, and blocking one does nothing to the other.

Training crawlers pull pages in bulk to build the next model. OpenAI’s is GPTBot, Anthropic’s is ClaudeBot, Google’s is controlled by the Google-Extended token, Apple’s by Applebot-Extended, and Common Crawl’s CCBot feeds most of the open datasets. Whatever they take ends up as weights, with no link back to you. There’s a case for blocking these and I don’t think it’s a silly one.

Retrieval bots are the other kind. When someone asks ChatGPT or Claude or Perplexity a question, the assistant runs a search, fetches a handful of pages right then, reads them, and writes an answer with citations. Those fetches arrive as OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User. Google’s AI Overviews use plain Googlebot, so there’s no separate switch for them at all. These are the bots that send a visitor back. If the aim is to be cited, these are the ones you’re optimising for, and it’s worth knowing that blocking GPTBot does nothing to ChatGPT’s search results in either direction.

This site’s robots.txt is Hugo’s default, which is allow everything:

User-agent: *
Allow: /
Sitemap: https://jared.lynskey.co.nz/sitemap.xml

I’ve thought about splitting it, letting the retrieval bots in and keeping the training ones out, and decided it wasn’t worth doing for a blog whose whole purpose is to be read. If I did want that, it would look like the block below. The reason to show it is the same lesson as in Cloudflare Served Googlebot an Empty Page: the list has to name bots, not vendors. There is no line you can write that says “OpenAI”.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

One thing to check if the site sits behind Cloudflare. Since the middle of 2025, new zones there block AI crawlers by default, and there’s a one-click toggle under the bot settings that does the same to existing ones. It blocks both kinds. This blog moved to Cloudflare Pages last week, so that was on my list. If you turned it on because “AI bots” sounded like something to keep out, you’ve also turned off the citations. Cloudflare has since started writing a Content-Signal line into the robots.txt files it manages, with separate search, ai-input and ai-train flags. Nobody is obliged to honour it, but it’s the right vocabulary: the same split as above, expressed as a preference rather than a wall.

What actually knocks on the door

#

The move to Cloudflare had a side effect: for the first time I can see every request by user agent. Here is what came in over the four days from 13 to 16 September 2026, after throwing out the fakes, which I’ll get to.

Two things stand out. The training crawlers read everything, twice, in the time it took the retrieval bots to read a few dozen pages. And the retrieval bots are not fetching at random: ChatGPT-User and Claude-User both went for the update server post, which is the most specific thing on the site, and the Japanese cold-start post got fetched in Japanese.

Now the fakes. Of 20 requests claiming to be GPTBot, 18 came from addresses outside OpenAI’s published ranges and asked for /.env, /.git/HEAD and /firebase-adminsdk.json. Fifteen of 36 “ChatGPT-User” requests were the same kind of thing. Most of the “PerplexityBot” traffic wanted /id_ed25519. A user agent is a string anyone can send, and vulnerability scanners have noticed that AI bot names get waved through. OpenAI, Anthropic and Perplexity all publish their IP ranges. If you’re going to treat a bot specially, in either direction, check the address, not the name. The 20, 59 and 8 above are the requests that came from the right addresses.

The page has to be readable without a browser

#

Retrieval bots fetch at answer time, with a timeout, and most of them don’t run JavaScript. A page that is an empty div until a bundle loads is a page they don’t read. Hugo writes finished HTML, so this blog gets that for free, and it’s the main reason I’ve never been tempted to move it to anything cleverer. The marketplace I run is the opposite case, a React Native web app, and getting crawlers a real document there took a dynamic-rendering branch and then a caching bug that quietly undid it. The Cloudflare post above is that story.

Speed counts the same way, and here the logs corrected me. I’d assumed a static site behind Cloudflare was served from the nearest edge. It isn’t. This blog is on Cloudflare Pages, the HTML goes out uncached, and from Seoul the response comes back from Los Angeles in about half a second. That’s still fine for a bot with a timeout of several seconds, but it isn’t the tens of milliseconds I’d have claimed, and I only know because I measured it. An assistant assembling an answer from six pages will drop the one that takes three seconds, and it doesn’t tell you it did.

Write so a paragraph can be lifted out

#

The way these systems use a page is not the way a reader does. They split it into chunks, score each chunk against the question, and quote the ones that score. A post that builds up to its point over eight hundred words scores badly on every chunk. A post where each section states its claim in the first sentence and then supports it scores on the section that matches.

That changed how I write here more than any technical thing did. The description in the front matter is now a summary of what the post says, not a hook to get you to read it, because the description is often the only part an assistant shows. Section headings say what the section concludes. Numbers go in the sentence with their units rather than in a chart. Dates are written out, because “last week” means nothing to a model reading the page in a year. And a post is about one thing.

The other half is having something worth quoting, and here the data is mixed. In the logs, the pages the answer-time bots fetch are the specific ones: Fly.io cold starts, the update server, iOS debugging. Naver’s crawler is called Yeti and Daum’s is Daumoa, a Fly.io restart is a thirty-second window for 504s, Cloudflare’s on-the-fly Brotli measured worse than gzip on my bundle. Those facts aren’t written down in many places, so a question that needs one of them lands here. But the people who click through from an assistant mostly land on the 2023 and 2024 posts about management, one or two sessions each. The specific posts get read by the machine. The general ones get the click. I don’t have a tidy explanation for that and I’d rather report it than invent one. The lever, if there is one, is still to write down the specific thing you found out this week that you couldn’t find when you searched for it, because that’s what the machines are demonstrably reading.

Say who wrote it, and keep saying the same thing

#

An assistant deciding whether to cite a page is doing something like what a careful reader does: who wrote this, when, and does it agree with everything else I’ve found? The mechanics of that on this site are mostly the theme’s doing. Every post carries Article structured data with an author, a published date and a modified date, and the author is a Person with links to the same GitHub and LinkedIn profiles. That’s a few lines in a template, and it’s the difference between “a page” and “a page by a named person with a history”.

The part that isn’t the theme’s doing is consistency. The about page, the llms.txt, the LinkedIn profile and the line at the top of every post all describe the same person with the same job history. When something changes, all of them change, or the one I forgot becomes the one that’s wrong. That sounds fussy, but one page saying I’m an engineering manager and another saying I’m freelance is exactly the kind of contradiction that makes a model hedge, or pick someone else.

The modified date has to be honest too. It changes when the content changes and I don’t touch it otherwise. Bumping it to look fresh is the kind of thing that works once and then gets your whole domain discounted.

llms.txt, and the fact that nothing reads it

#

llms.txt is a proposal from September 2024 for a Markdown file at the site root that gives a language model a summary of the site and a curated list of links. I added one in February this year. It’s a paragraph about who I am, then sections for management, infrastructure, mobile and community, with one line per post: the title, the URL, and a sentence saying what the post concludes. It reads like a table of contents written for someone in a hurry.

The caveat is that nothing reads it. In the four days of logs I have, the only requests for /llms.txt were my own. Common Crawl has captured 76 pages of this site in its August 2026 crawl and never that file. No major assistant has said it reads it. So it is not why anything here gets cited, and I’d stop short of recommending it to anyone on those grounds. I keep it for two reasons. It cost an hour, and writing one sentence per post that says what the post actually concludes was worth more than the file, because it showed me which posts didn’t conclude anything. It’s also the one page on the site that tells a machine what the site is for, and I’d rather have that and not need it. It has drifted since, because I write it by hand and forget, which is the argument for generating it from the front matter as a Hugo output format instead. That’s on the list.

Feeds, sitemaps and five languages

#

The boring plumbing still counts. The sitemap is how a crawler finds a new post the day it goes up. RSS is how it learns a post changed. Both are Hugo defaults and I’ve never had to think about them. ClaudeBot read the sitemap 36 times in two days, which is how it found 1,380 pages so quickly.

The languages are the part I didn’t expect to matter. Everything here is in English, Japanese, Korean, Simplified Chinese and Vietnamese, with alternate links so a crawler knows they’re the same post. The crawlers treat them as five sites: ClaudeBot’s crawl split almost evenly across the five, and of the 73 post pages Common Crawl took in August, only four were the English ones. Assistants answer in the language they’re asked in and lean towards sources in that language, so a Korean question about Fly.io cold starts can be answered from the Korean page rather than from a translation of the English one done on the fly, and the Japanese page is the one ChatGPT actually fetched. The condition is that the translation reads as if it were written in that language. The pass I did on that earlier this year, with native speakers checking, was for human readers, but it’s also what makes the Vietnamese page a source rather than a curiosity.

What I can measure

#

More than I thought, once I stopped guessing. Cloudflare’s free plan gives request logs by user agent and path, one day per query, which is where the bot table came from; referrers need the paid plan. Google Analytics has the referrers, and pulling them through its API rather than squinting at a dashboard is what produced the numbers at the top. Together they show the shape of it. In four days the retrieval bots fetched about ninety pages from OpenAI and Anthropic addresses. In the last month, two people arrived from ChatGPT. Read a lot, cited sometimes, clicked rarely. There’s still no way I know of to see the answers that used a page and didn’t get clicked. The most useful check is still the manual one: ask the assistant the question your post answers and see whose page it quotes.

If I had to keep one thing from all of this, it’s the writing rule. The robots.txt, the structured data and the llms.txt are an afternoon’s work and then they’re done. Writing so each section says its claim first, and writing about the specific thing rather than the general one, is a habit, and it’s the habit that decides whether a page is worth quoting at all. The second thing is to measure before you claim. Being trusted by a machine turns out to look a lot like being trusted by a person: say who you are, say what you know, and say it the same way every time.