Cloudflare Served Googlebot an Empty Page

My marketplace is a React Native Web app exported by Expo. Every listing page is one script tag and an empty div until JavaScript runs. That is fine for people and fatal for search: Google can sometimes execute the JS, Bing will not, and Naver’s crawler — the one that matters most for a Korean-community site in New Zealand — certainly will not.

This is the story of making those pages visible, and then of the caching rule that undid it in a way I did not see coming.

Dynamic rendering, which is not cloaking

#

The fix is old and Google-sanctioned: detect a crawler, and serve it server-rendered HTML containing the same content the SPA would have painted. Not different content. The same content, earlier.

So /tabs/used-good-view?id=... branches. A browser gets the Expo shell. A crawler gets a real document: an <h1>, a <dl> of facts (price, condition, category, location, availability, photo count, listing date), an <img> per photo with generated alt text, a body excerpt, and links to eight sibling listings so the page is not an orphan.

That last part mattered more than I expected. Before those internal links existed, roughly a thousand listing pages sat with nothing pointing at them, and Search Console showed the crawl settling to seven to eleven requests a day. A crawler that cannot find your pages does not care how good they are.

The trap: a vendor’s name is not their bot’s name

#

Crawler detection is a user-agent match, and here is where it bit.

Korean platforms crawl under names that look nothing like the company. Naver’s crawler is Yeti. Daum’s is Daumoa. Kakao’s link scraper is kakaotalk-scrap. None of them is the vendor name.

Meanwhile those same vendors ship apps with embedded browsers, and those send a real browser user-agent with the app’s name appended. KakaoTalk’s in-app browser says KAKAOTALK/10.5.1. Kakao’s scraper says kakaotalk-scrap/1.0.

Match on kakao and you match both. Which is exactly what I did, and so every KakaoTalk, Naver-app and Daum-app user who opened a link got the crawler page: no JavaScript, no navigation, no login. A perfectly good document, and completely useless to a human.

The fix is an ordering rule, and it is worth stating as one:

def is_crawler(request):
    """An in-app browser is never a crawler, even when its UA carries the
    same vendor name the vendor's bot does — so that check comes first."""
    ua = request.META.get("HTTP_USER_AGENT", "")
    if not ua or _INAPP_BROWSER_RE.search(ua):
        return False
    return bool(_CRAWLER_RE.search(ua))

In-app browsers are checked first and win. And every crawler token has to name a bot, never a vendor:

# Korean engines. Naver crawls as Yeti; Daum as Daumoa; Kakao's link-preview
# scraper as kakaotalk-scrap. None of the three is the bare vendor name.
r"yeti/|naverbot|daumoa|kakaotalk-scrap|kakao[a-z]*bot|"

Then I put Cloudflare in front of it

#

The origin is a single shared-cpu machine in Sydney that sustains twenty to thirty requests a second. Googlebot has about 995 URLs to walk. Every one of them was reaching Sydney, because cf-cache-status read DYNAMIC on all of them: Cloudflare does not cache HTML unless a rule tells it to. About 16% of the zone was cached, and none of the part that mattered.

So: a cache rule on the crawler pages. Straightforward. Except that a listing URL does not have one body. It has three.

• A crawler asking for HTML gets the server-rendered page.
• A human gets the Expo shell.
• An agent sending Accept: text/markdown gets markdown, built from the database, because converting an empty HTML shell would produce an empty document.

The origin says Vary: Accept and Vary on the user-agent would be hopeless anyway. And here is the thing I did not know:

Cloudflare ignores Vary on HTML. All three bodies share one cache key.

Whichever body arrives first wins that URL for every later visitor in that edge location, for the hour the origin’s s-maxage allows, plus a day of stale-while-revalidate.

I got to observe this in production in both directions:

• A Naver-app user warmed a listing URL with the crawler page, and plain iPhone Safari then got that page on a HIT. A human staring at a JavaScript-less document.
• A human warmed a URL with the Expo shell, and Googlebot got the empty shell. The exact failure the whole SEO effort existed to prevent, now served faster and from the edge.

The rule that fixes it by narrowing

#

You cannot make Cloudflare respect Vary on HTML. What you can do is make sure only one of those three bodies is ever eligible to enter the cache:

(starts_with(http.request.uri.path, "/tabs/")
 and not any(http.request.headers["accept"][*] contains "text/markdown")
 and (lower(http.user_agent) contains "googlebot" or ...)
 and not (lower(http.user_agent) contains "kakaotalk/" or ...))

Match crawler user-agents only, and exclude markdown requests. Now a human request matches no rule at all, so it reads DYNAMIC and goes to the origin as before. Only crawler requests read or write the cache, and the only body that can ever be in there is the crawler one.

Edge TTL is respect_origin, so the lifetime stays owned by the code that renders the page rather than being pinned in a dashboard.

One operational detail that is easy to miss: narrowing a rule does not evict what is already in the cache. You have to purge, or you keep serving the poisoned entries you just stopped creating.

The two lists, and why their errors are not symmetric

#

The rule needs a list of crawler user-agents, and Django already has one. Two copies of the same knowledge in two systems is usually a smell, but here the interesting part is what happens when they drift, because it is not symmetric:

• The bot list in Cloudflare must stay a subset of the one in Django. A token Cloudflare thinks is a bot but Django does not would cache an SPA shell under the crawler-only key, and then serve that to a real crawler.
• The in-app browser exclusion list must stay a superset of Django’s. Excluding too much only costs you a cache hit.

Get either one wrong in the safe direction and you lose some caching. Get the first one wrong in the other direction and you are back to serving Googlebot an empty page. That asymmetry is why the lists use contains rather than a regex, incidentally: matches needs a Business plan, and contains is expressive enough for a list that is allowed to be approximate in a known-safe direction.

An invariant that fails the build instead of the page

#

The crawler pages embed presigned image URLs, valid for seven days. The pages themselves are cached at the edge for an hour, plus a day of stale-while-revalidate.

Those two numbers are related, and if the cache lifetime ever exceeds the image lifetime, the edge starts serving pages whose every photo 403s. Nothing would alert on it; the HTML is fine.

So the module that owns those constants refuses to import if the relationship breaks:

CRAWLER_IMAGE_TTL = 7 days
# raises at import if s-maxage + stale-while-revalidate >= CRAWLER_IMAGE_TTL

A constraint between two constants in different files is invisible in code review and obvious at process start. Put it where the process starts.

Two things Cloudflare did that I had to undo

#

Vary: Origin kept the bundle out of the cache. django-cors-headers stamps Vary: Origin on every response it touches, and Cloudflare declines to cache anything that varies on more than Accept-Encoding. That quietly excluded the 1.6 MB JavaScript bundle — the single biggest thing on the critical path — from the edge entirely. The fix is a middleware placed first in the list, so it runs last on the way out, rewriting Vary for /_expo/ asset paths.

Compression at the edge was worse than compression at the origin. Letting Cloudflare do Brotli on the fly produced 2,041,742 bytes where the origin’s own gzip produced 1,658,115. Twenty-three percent larger, for the privilege of not thinking about it. On-the-fly compression optimises for CPU, not for ratio; precompressed assets from the origin win.

Telling search engines without waiting to be crawled

#

Two protocols, and they cover disappointingly little between them.

IndexNow takes a ping and covers Bing, Yandex and Seznam. Google and Naver do not participate. It runs on post_save in a daemon thread with a five-second timeout, skips sold and closed listings, and is a no-op without a key. A network failure logs a warning and never fails the save that triggered it, which is the only acceptable behaviour for a best-effort side channel.

Google’s Indexing API accepts exactly two schema types, and one of them is JobPosting, which I have. Live jobs send URL_UPDATED, closed ones send URL_DELETED. That deletion is only honest because closed listings also emit noindex, and the code says so, so that whoever removes the noindex finds out what else they broke.

Two things cost me time there. The quota is 200 URLs a day, and a scraper import run fires the signal hundreds of times — so imported listings are skipped, or one import spends the entire day’s budget. And the endpoint is urlNotifications:publish, with a colon. Use a slash and Google returns a plain HTML 404, which looks exactly like a disabled API.

What the structured data taught me

#

Three rules, each learned by having Search Console complain:

1. Omit what you cannot parse. Salaries arrive as free text. I used to emit the raw string as baseSalary.value, which is invalid, and flagged every job for a missing unit. Now an unparseable salary is simply absent.
2. Never fabricate a field to satisfy a schema. Job ads rarely carry a street address. The temptation is to invent something plausible for PostalAddress. A fabricated address on a job ad is worse than an incomplete one, so only locality and country go in.
3. Expiry is not optional. Google drops a JobPosting with no validThrough, so one is derived, with a floor so a stale listing does not claim to have expired yesterday.

And a fourth, about sitemaps: do not claim changefreq: daily for everything. Asking Googlebot to re-crawl a thousand listings every day reads as an overloaded host, and it responds by cutting the crawl rate for the whole site. Derive it from how recently the thing actually changed.

The sitemap also has a quality floor: forty characters of title and body combined. Search Console had reported around a thousand listings as “Crawled – currently not indexed,” and one used-goods page’s entire unique content was a one-word title and a phone number. Forty characters is deliberately low, because 471 of 484 listings are in Korean and Korean says more per character; an English-tuned threshold would have dropped half the marketplace. Thin listings stay crawlable through the index pages, and re-enter the sitemap automatically when someone edits them.

What I would take from this

#

• Serving different bodies at one URL is a cache-key problem before it is anything else. Work out what your CDN keys on, and assume it ignores Vary until you have proven otherwise on that exact content type.
• When two systems must agree on a list, work out which direction of error is safe and write that down next to both copies. “These can drift, and here is the only way they are allowed to” beats pretending they cannot.
• Match bots by bot name, never by vendor name. Especially outside the Anglosphere, where the vendor also ships the browser your users are holding.
• Put invariants between distant constants somewhere that fails loudly. Import time is free and nobody can skip it.