<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Caching · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/caching/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Sat, 07 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/caching/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>React Native Web SEO: When Cloudflare Served Googlebot an Empty Page</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-cloudflare-crawlers-cache/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-cloudflare-crawlers-cache/</guid>
      <pubDate>Wed, 09 Sep 2026 04:00:00 +1200</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Cloudflare</category>
      <category>SEO</category>
      <category>Django</category>
      <category>Caching</category>
      <category>Crawlers</category>
      <category>Naver</category>
      <category>React Native</category>
      <description>How I made a React Native Web app visible to Google and Naver&#39;s crawler with dynamic rendering, and the Cloudflare cache rule that served Googlebot the empty SPA shell.</description>
      <content:encoded><![CDATA[<p>If your React or React Native Web site is a single-page app, Naver&rsquo;s crawler (Yeti) and Bing won&rsquo;t run its JavaScript, and Google only sometimes does. The fix is dynamic rendering: detect crawlers by user-agent and serve them server-rendered HTML with the same content the app would paint. The Cloudflare mistake to avoid is caching that URL for everyone. Cloudflare ignores <code>Vary</code> on HTML, so whichever body arrives first is served to every later visitor, and Googlebot can end up with the empty shell; cache only crawler requests, and purge after you narrow the rule.</p>
<p>My marketplace app is a single-page app built with React Native Web. Every listing page is one script tag and an empty div until JavaScript runs. That is fine for people and fatal for search: Google can sometimes execute the JS, Bing will not, and Naver&rsquo;s crawler — the one that matters most when nearly all your listings are in Korean — certainly will not.</p>
<p>This is the story of making those pages visible, and then of the caching rule that undid it in a way I did not see coming.</p>

<h2 class="relative group">Dynamic rendering, which is not cloaking
    <div id="dynamic-rendering-which-is-not-cloaking" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#dynamic-rendering-which-is-not-cloaking" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The fix is old and Google-sanctioned: detect a crawler, and serve it server-rendered HTML containing the same content the SPA would have painted. Not different content. The same content, earlier.</p>
<p>So a listing URL like <code>/listings/view?id=...</code> branches. A browser gets the SPA shell. A crawler gets a real document: a heading, the listing&rsquo;s key facts, an image per photo with alt text, a body excerpt, and links to a handful of related listings so the page is not an orphan.</p>
<p>That last part mattered more than I expected. Before those internal links existed, roughly a thousand listing pages sat with nothing pointing at them, and Search Console showed the crawl settling to seven to eleven requests a day. A crawler that cannot find your pages does not care how good they are.</p>

<h2 class="relative group">The trap: a vendor&rsquo;s name is not their bot&rsquo;s name
    <div id="the-trap-a-vendors-name-is-not-their-bots-name" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-trap-a-vendors-name-is-not-their-bots-name" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Crawler detection is a user-agent match, and here is where it bit.</p>
<p>Korean platforms crawl under names that look nothing like the company. Naver&rsquo;s crawler is <strong>Yeti</strong>. Daum&rsquo;s is <strong>Daumoa</strong>. Kakao&rsquo;s link scraper is <strong>kakaotalk-scrap</strong>. None of them is the vendor name.</p>
<p>Meanwhile those same vendors ship apps with embedded browsers, and those send a real browser user-agent with the app&rsquo;s name appended. KakaoTalk&rsquo;s in-app browser says <code>KAKAOTALK/10.5.1</code>. Kakao&rsquo;s scraper says <code>kakaotalk-scrap/1.0</code>.</p>
<p>Match on <code>kakao</code> and you match both. Which is exactly what I did, and so every KakaoTalk, Naver-app and Daum-app user who opened a link got the crawler page: no JavaScript, no navigation, no login. A perfectly good document, and completely useless to a human.</p>
<p>The fix is an ordering rule, and it is worth stating as one:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">re</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># In-app browsers append the app&#39;s name to an ordinary browser UA.</span>
</span></span><span class="line"><span class="cl"><span class="n">INAPP_BROWSER_RE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;kakaotalk/|naver\(inapp|daumapps&#34;</span><span class="p">,</span> <span class="n">re</span><span class="o">.</span><span class="n">I</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Name the bot, never the vendor: Naver crawls as Yeti, Daum as Daumoa,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Kakao&#39;s link-preview scraper as kakaotalk-scrap.</span>
</span></span><span class="line"><span class="cl"><span class="n">CRAWLER_RE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;googlebot|bingbot|yeti/|daumoa|kakaotalk-scrap&#34;</span><span class="p">,</span> <span class="n">re</span><span class="o">.</span><span class="n">I</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">is_crawler</span><span class="p">(</span><span class="n">user_agent</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># An in-app browser is never a crawler, so that check comes first.</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">user_agent</span> <span class="ow">or</span> <span class="n">INAPP_BROWSER_RE</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">user_agent</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">CRAWLER_RE</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">user_agent</span><span class="p">))</span></span></span></code></pre></div></div>
<p>In-app browsers are checked first and win. And every crawler token has to name a <em>bot</em>, never a vendor.</p>

<h2 class="relative group">Then I put Cloudflare in front of it
    <div id="then-i-put-cloudflare-in-front-of-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#then-i-put-cloudflare-in-front-of-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The origin is a single shared-cpu machine in Sydney that sustains twenty to thirty requests a second. Googlebot has about 995 URLs to walk. Every one of them was reaching Sydney, because <code>cf-cache-status</code> read <code>DYNAMIC</code> on all of them: <strong>Cloudflare does not cache HTML unless a rule tells it to.</strong> About 16% of the zone was cached, and none of the part that mattered.</p>
<p>So: a cache rule on the crawler pages. Straightforward. Except that a listing URL does not have one body. It has three.</p>
<ul>
<li>A crawler asking for HTML gets the server-rendered page.</li>
<li>A human gets the SPA shell.</li>
<li>An agent sending <code>Accept: text/markdown</code> gets markdown, built from the database, because converting an empty HTML shell would produce an empty document.</li>
</ul>
<p>The origin says <code>Vary: Accept</code> and <code>Vary</code> on the user-agent would be hopeless anyway. And here is the thing I did not know:</p>
<blockquote><p><strong>Cloudflare ignores <code>Vary</code> on HTML.</strong> All three bodies share one cache key.</p>
</blockquote><p>Whichever body arrives first wins that URL for every later visitor in that edge location, for the hour the origin&rsquo;s <code>s-maxage</code> allows, plus a day of <code>stale-while-revalidate</code>.</p>
<p>I got to observe this in production in both directions:</p>
<ul>
<li>A Naver-app user warmed a listing URL with the crawler page, and plain iPhone Safari then got that page on a HIT. A human staring at a JavaScript-less document.</li>
<li>A human warmed a URL with the SPA shell, and Googlebot got the empty shell. The exact failure the whole SEO effort existed to prevent, now served faster and from the edge.</li>
</ul>

<h2 class="relative group">The rule that fixes it by narrowing
    <div id="the-rule-that-fixes-it-by-narrowing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-rule-that-fixes-it-by-narrowing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>You cannot make Cloudflare respect <code>Vary</code> on HTML. What you can do is make sure only one of those three bodies is ever eligible to enter the cache:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">(starts_with(http.request.uri.path, &#34;/listings/&#34;)
</span></span><span class="line"><span class="cl"> and not any(http.request.headers[&#34;accept&#34;][*] contains &#34;text/markdown&#34;)
</span></span><span class="line"><span class="cl"> and (lower(http.user_agent) contains &#34;googlebot&#34; or ...)
</span></span><span class="line"><span class="cl"> and not (lower(http.user_agent) contains &#34;kakaotalk/&#34; or ...))</span></span></code></pre></div></div>
<p>Match crawler user-agents only, and exclude markdown requests. Now a human request matches no rule at all, so it reads <code>DYNAMIC</code> and goes to the origin as before. Only crawler requests read or write the cache, and the only body that can ever be in there is the crawler one.</p>
<p>Edge TTL is <code>respect_origin</code>, so the lifetime stays owned by the code that renders the page rather than being pinned in a dashboard.</p>
<p>One operational detail that is easy to miss: <strong>narrowing a rule does not evict what is already in the cache.</strong> You have to purge, or you keep serving the poisoned entries you just stopped creating.</p>

<h2 class="relative group">The two lists, and why their errors are not symmetric
    <div id="the-two-lists-and-why-their-errors-are-not-symmetric" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-two-lists-and-why-their-errors-are-not-symmetric" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The rule needs a list of crawler user-agents, and Django already has one. Two copies of the same knowledge in two systems is usually a smell, but here the interesting part is what happens when they drift, because it is not symmetric:</p>
<ul>
<li>The <strong>bot list in Cloudflare must stay a subset</strong> of the one in Django. A token Cloudflare thinks is a bot but Django does not would cache an SPA shell under the crawler-only key, and then serve that to a real crawler.</li>
<li>The <strong>in-app browser exclusion list must stay a superset</strong> of Django&rsquo;s. Excluding too much only costs you a cache hit.</li>
</ul>
<p>Get either one wrong in the safe direction and you lose some caching. Get the first one wrong in the other direction and you are back to serving Googlebot an empty page. That asymmetry is why the lists use <code>contains</code> rather than a regex, incidentally: <code>matches</code> needs a Business plan, and <code>contains</code> is expressive enough for a list that is allowed to be approximate in a known-safe direction.</p>

<h2 class="relative group">An invariant that fails the build instead of the page
    <div id="an-invariant-that-fails-the-build-instead-of-the-page" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#an-invariant-that-fails-the-build-instead-of-the-page" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The crawler pages embed presigned image URLs, valid for seven days. The pages themselves are cached at the edge for an hour, plus a day of stale-while-revalidate.</p>
<p>Those two numbers are related, and if the cache lifetime ever exceeds the image lifetime, the edge starts serving pages whose every photo 403s. Nothing would alert on it; the HTML is fine.</p>
<p>So the code that defines those numbers refuses to import if the relationship breaks:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">CRAWLER_IMAGE_TTL</span> <span class="o">=</span> <span class="mi">7</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span>  <span class="c1"># presigned image URLs</span>
</span></span><span class="line"><span class="cl"><span class="n">EDGE_S_MAXAGE</span> <span class="o">=</span> <span class="mi">3600</span>
</span></span><span class="line"><span class="cl"><span class="n">EDGE_STALE_WHILE_REVALIDATE</span> <span class="o">=</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Fail at import time if the edge cache could outlive the image URLs.</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="n">EDGE_S_MAXAGE</span> <span class="o">+</span> <span class="n">EDGE_STALE_WHILE_REVALIDATE</span> <span class="o">&gt;=</span> <span class="n">CRAWLER_IMAGE_TTL</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s2">&#34;edge cache lifetime must stay below the image URL lifetime&#34;</span><span class="p">)</span></span></span></code></pre></div></div>
<p>A constraint between two constants in different files is invisible in code review and obvious at process start. Put it where the process starts.</p>

<h2 class="relative group">Two things Cloudflare did that I had to undo
    <div id="two-things-cloudflare-did-that-i-had-to-undo" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#two-things-cloudflare-did-that-i-had-to-undo" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong><code>Vary: Origin</code> kept the bundle out of the cache.</strong> <code>django-cors-headers</code> stamps <code>Vary: Origin</code> on every response it touches, and Cloudflare declines to cache anything that varies on more than <code>Accept-Encoding</code>. That quietly excluded the 1.6 MB JavaScript bundle — the single biggest thing on the critical path — from the edge entirely. The fix is a middleware placed first in the list, so it runs last on the way out, rewriting <code>Vary</code> on the static asset paths.</p>
<p><strong>Compression at the edge was worse than compression at the origin.</strong> Letting Cloudflare do Brotli on the fly produced 2,041,742 bytes where the origin&rsquo;s own gzip produced 1,658,115. Twenty-three percent larger, for the privilege of not thinking about it. On-the-fly compression optimises for CPU, not for ratio; precompressed assets from the origin win.</p>

<h2 class="relative group">Telling search engines without waiting to be crawled
    <div id="telling-search-engines-without-waiting-to-be-crawled" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#telling-search-engines-without-waiting-to-be-crawled" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Two protocols, and they cover disappointingly little between them.</p>
<p><strong>IndexNow</strong> takes a ping and covers Bing, Yandex and Seznam. Google and Naver do not participate. I ping it in the background whenever a listing is saved, with a short timeout, and skip listings that are sold or closed. A network failure logs a warning and never fails the save that triggered it, which is the only acceptable behaviour for a best-effort side channel.</p>
<p><strong>Google&rsquo;s Indexing API</strong> accepts exactly two schema types, and one of them is <code>JobPosting</code>, which I have. Live jobs send <code>URL_UPDATED</code>, closed ones send <code>URL_DELETED</code>. That deletion is only honest because closed listings also emit <code>noindex</code>, and the code says so, so that whoever removes the noindex finds out what else they broke.</p>
<p>Two things cost me time there. The quota is 200 URLs a day, and a bulk import of listings would fire it hundreds of times — so imported listings are skipped, or one import spends the entire day&rsquo;s budget. And the endpoint is <code>urlNotifications:publish</code>, with a colon. Use a slash and Google returns a plain HTML 404, which looks exactly like a disabled API.</p>

<h2 class="relative group">What the structured data taught me
    <div id="what-the-structured-data-taught-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-the-structured-data-taught-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Three rules, each learned by having Search Console complain:</p>
<ol>
<li><strong>Omit what you cannot parse.</strong> Salaries arrive as free text. I used to emit the raw string as <code>baseSalary.value</code>, which is invalid, and flagged every job for a missing unit. Now an unparseable salary is simply absent.</li>
<li><strong>Never fabricate a field to satisfy a schema.</strong> Job ads rarely carry a street address. The temptation is to invent something plausible for <code>PostalAddress</code>. A fabricated address on a job ad is worse than an incomplete one, so only locality and country go in.</li>
<li><strong>Expiry is not optional.</strong> Google drops a <code>JobPosting</code> with no <code>validThrough</code>, so one is derived, with a floor so a stale listing does not claim to have expired yesterday.</li>
</ol>
<p>And a fourth, about sitemaps: <strong>do not claim <code>changefreq: daily</code> for everything.</strong> Asking Googlebot to re-crawl a thousand listings every day reads as an overloaded host, and it responds by cutting the crawl rate for the whole site. Derive it from how recently the thing actually changed.</p>
<p>The sitemap also has a quality floor: forty characters of title and body combined. Search Console had reported around a thousand listings as &ldquo;Crawled – currently not indexed,&rdquo; and one used-goods page&rsquo;s entire unique content was a one-word title and a phone number. Forty characters is deliberately low, because 471 of 484 listings are in Korean and Korean says more per character; an English-tuned threshold would have dropped half the marketplace. Thin listings stay crawlable through the index pages, and re-enter the sitemap automatically when someone edits them.</p>

<h2 class="relative group">What I would take from this
    <div id="what-i-would-take-from-this" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-would-take-from-this" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li><strong>Serving different bodies at one URL is a cache-key problem before it is anything else.</strong> Work out what your CDN keys on, and assume it ignores <code>Vary</code> until you have proven otherwise on that exact content type.</li>
<li><strong>When two systems must agree on a list, work out which direction of error is safe</strong> and write that down next to both copies. &ldquo;These can drift, and here is the only way they are allowed to&rdquo; beats pretending they cannot.</li>
<li><strong>Match bots by bot name, never by vendor name.</strong> Especially outside the Anglosphere, where the vendor also ships the browser your users are holding.</li>
<li><strong>Put invariants between distant constants somewhere that fails loudly.</strong> Import time is free and nobody can skip it.</li>
</ul>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-cloudflare-crawlers-cache/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
