Cloudflare 504 Errors That Aren't Real: Early Hints and HTTP 103

In September I wrote about a Cloudflare dashboard that said 22% of my requests were failing while every health check stayed green. Most of those 504s turned out to come from Cloudflare’s Early Hints feature rather than from my users, so I switched Early Hints off and moved on.

In November I opened the same zone and found another 2,600 504s in a single day. Early Hints was on again. This time I wanted to know exactly what those 504s are, and whether the right fix was to turn the feature off a second time or to make it actually do its job.

The zone fronts one Fly.io origin that serves three things: a Django API, an Expo web app, and a Hugo marketing site. Keep the API in mind. It has a real capacity story of its own, and it’s the reason this distinction matters.

The short answer

#

• Those 504s are written by Cloudflare’s Early Hints cache, not by your visitors. In Cloudflare’s logs they carry RequestSource = earlyHintsCache.
• A 504 there means a cache miss: Cloudflare had no hints stored for that URL. A hit is logged as a 2xx.
• They never reach your origin. OriginResponseStatus is 0.
• Early Hints runs on the Link HTTP response header. <link rel="preload"> tags in your HTML don’t feed it.
• Send the header, and those misses turn into hits and real 103 Early Hints responses on the pages that have something to preload.

How to tell a fake Cloudflare 504 from a real one

#

If you use Logpush or Instant Logs, look at two fields before anything else. RequestSource is eyeball for a real visitor’s request and earlyHintsCache for the Early Hints lookups. OriginResponseStatus is 0 when the origin was never contacted.

In the GraphQL Analytics API the same split is the requestSource dimension. This is the query I now run before believing any 5xx spike:

query ($zone: String!, $since: Time!) {
  viewer {
    zones(filter: { zoneTag: $zone }) {
      httpRequestsAdaptiveGroups(
        limit: 20
        filter: { datetime_geq: $since, edgeResponseStatus: 504 }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { requestSource userAgent clientRequestPath }
      }
    }
  }
}

Grouping by user agent tells the same story. In September the Early Hints lookups showed up as bastion early hints and nginx-ssl early hints. In November they were fl2 early hints and nginx-ssl early hints (FL2 is the newer generation of Cloudflare’s proxy). Across 24 hours, all 2,631 of the zone’s 504s came from those user agents. None came from a browser, the app, curl or a search crawler.

A real 504 that a visitor actually saw came from your side of Cloudflare. A proxy or load balancer in front of your application gave up waiting: nginx, a cloud load balancer, or a platform proxy waiting for a cold machine to boot. Cloudflare’s own timeouts use different codes. 522 means it couldn’t connect to your origin, and 524 means it connected but got no response within 100 seconds (the default on plans below Enterprise).

So if you landed here searching for how to raise Cloudflare’s timeout because of 504s: the timeout behind a 504 is almost never Cloudflare’s. Check proxy_read_timeout in nginx, your load balancer’s idle timeout, or how long your platform takes to wake a machine. A useful clue: if requests fail at a round number well under 100 seconds, such as 30 or 60, it’s your own proxy’s limit, not Cloudflare’s. (Only Enterprise plans can raise Cloudflare’s 100-second limit, up to 6,000 seconds, and Cloudflare itself treats that as a stopgap.)

The same week, the API had real 504s

#

This is why the split matters. A few days earlier the same API had genuinely fallen over. It was running on Fly.io’s shared-cpu-1x machines, which are burst-credited: under New Zealand peak traffic they ran out of burst and were throttled to a sustained ceiling of roughly 20 to 30 requests a second. Response times collapsed and visitors got real 504s at the edge. Moving to shared-cpu-2x fixed it.

Then the 504 chart filled up again, this time with Early Hints misses. Read without the requestSource split, that chart says the bigger machines didn’t help. In fact there were two separate populations of 504s on the same zone in the same week, and only one of them ever reached a person. Don’t make capacity decisions from a 5xx chart you haven’t split by request source.

What those 504s actually are

#

On a zone with Early Hints enabled, every request gets an extra internal lookup: Cloudflare checking its own cache for a set of hints to send. That lookup is logged as its own line, with RequestSource = earlyHintsCache. It isn’t an end-user request, and it is never sent to your origin.

The status on that line is the result of the cache lookup, not an error:

• 2xx is a hit. Cloudflare had Link headers stored for that URL and sent them to the browser as a 103. On my zone, hits show up as both 200 and 204.
• 504 is a miss. Nothing was stored, so there was nothing to send early.

If your origin never sends the header Early Hints runs on, every lookup is a miss and every request writes a 504 line. In the three hours before I changed anything, the zone logged 173 Early Hints lookups and all 173 were misses. Since the feature had been switched on, it had not served a single hint.

One correction to my September post: I described these as probes that “get no response from the Fly origin”. That isn’t quite right. They never go to the origin at all. The 504 records a cache miss, not a timeout.

Is HTTP 103 an error?

#

No. Status codes in the 1xx range are informational: interim responses a server can send before the real one. 103 Early Hints, defined in RFC 8297, carries Link headers so the browser can start fetching critical CSS, JavaScript and fonts, or open connections to other origins, while the server is still building the page. The final 200 follows on the same request.

Support is uneven but safe. Chromium-based browsers (Chrome since version 103, and Edge) and current Firefox act on both preload and preconnect hints. Safari uses only preconnect. Browsers process 103 only over HTTP/2 and HTTP/3. A browser that ignores it simply waits for the 200 as it always did, so a 103 is a head start for most visitors and a no-op for the rest.

Why mine was empty: the Link header vs <link rel=preload>

#

This is the part that cost me the most time, because it looks like it should already work.

Early Hints caches exactly one thing: the HTTP Link response header with a rel=preload or rel=preconnect relation. Cloudflare reads it from your normal responses (status 200, 301 or 302, on HTML or extensionless URLs), stores it, and replays it to the next visitor as a 103 before your full response is ready.

My pages already had preloads, but as tags in the HTML <head>:

<link rel="preload" href="/_expo/static/css/global-7d77bd.css" as="style">
<link rel="preload" href="/_expo/static/js/web/entry-fd7fa4.js" as="script">
<link rel="preload" href="/fonts/manrope-latin.woff2" as="font" type="font/woff2" crossorigin>

Those tags still do something: the browser acts on them. But it only sees them after it has received the HTML and started parsing it, which is exactly the moment Early Hints exists to beat. The Expo export and Hugo both write these tags for you, and neither sets the header. So the feature was enabled, the preloads were declared, and the two never met.

The fix: send the header you already have the data for

#

The cleanest fix doesn’t involve maintaining a hand-written list of critical assets that goes stale every time a bundle hash changes. Each page already declares what it wants preloaded, in those tags. So mirror them into a Link header when you serve the page:

import re

PRELOAD = re.compile(rb"""<link\b[^>]*?\brel=["']?preload["']?[^>]*?>""", re.I)


def attr(tag: bytes, name: bytes) -> bytes | None:
    # A real attribute: preceded by whitespace, followed by =, space, / or >.
    m = re.search(
        rb"(?<=\s)" + re.escape(name)
        + rb"""(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>]+)))?(?=[\s/>])""",
        tag,
        re.I,
    )
    if not m:
        return None
    return next((g for g in m.groups() if g is not None), b"")


def link_header(html: bytes, limit: int = 12) -> str | None:
    parts, seen = [], set()
    for tag in PRELOAD.findall(html):
        href = attr(tag, b"href")
        if not href or href in seen:
            continue
        seen.add(href)
        seg = f"<{href.decode()}>; rel=preload"
        if as_ := attr(tag, b"as"):
            seg += f"; as={as_.decode()}"
        if as_ == b"font" or attr(tag, b"crossorigin") is not None:
            seg += "; crossorigin"
        parts.append(seg)
        if len(parts) == limit:
            break
    return ", ".join(parts) or None

And on the HTML response, in Django:

response = HttpResponse(html, content_type="text/html")
if (link := link_header(html)) and "Link" not in response:
    response["Link"] = link

Two details bit me, and they’re worth keeping:

• Anchor the attribute parser. A naive search for as happily matches the “as” inside href="/assets/...", and your font preloads silently lose as=font. Require the name to start after whitespace and end at =, a space, / or >. Handle both quoted and unquoted attributes too, because static site generators like Hugo emit href=/fonts/x.woff2 as=font crossorigin with no quotes. A unit test caught this one for me.
• Keep crossorigin on fonts. A font preload without it is treated as a different request, so the font gets downloaded twice and the page gets slower, not faster.

You don’t write the 103 yourself. There’s no ASGI, Granian or uvicorn interim-response code to add. Cloudflare builds the 103 from your header; your only job is to send the header.

On other stacks

#

The rule is the same everywhere: put a Link header on the HTML response and let Cloudflare do the rest.

• nginx, on the location that serves your HTML:

  location = / {
      add_header Link "</css/app.css>; rel=preload; as=style" always;
      add_header Link "</fonts/inter.woff2>; rel=preload; as=font; crossorigin" always;
  }

• Laravel with Vite already has middleware for this, AddLinkHeadersForPreloadedAssets, which turns Vite’s preloads into Link headers. Two issues reported against it are the same traps as above: its modulepreload hints have gone out without crossorigin, so browsers can discard them and download the script twice, and on large pages the header has grown past Apache’s roughly 8 KB header limit. Check both on your version.

• Anything else: set one Link header, comma-separated, or several Link headers. Either works.

What to hint is where people overdo it. Hint the render-critical CSS, the entry script and the one or two fonts above the fold, not every chunk. A long list makes the browser fetch things it doesn’t need yet, ahead of things it does. I cap mine at twelve, and only because the Expo export already curates its preload list.

If you’re not behind Cloudflare, or your CDN doesn’t do Early Hints, the Link header still helps a little: the browser can start those fetches as soon as the response headers arrive, before it parses the body. But no 103 is sent unless your server or CDN emits one.

How to check it’s working

#

curl shows the 103 directly. Ask for HTTP/2 and keep only the status lines and Link headers:

$ curl -sv --http2 -o /dev/null https://example.com/register 2>&1 | grep -E '^< (HTTP|link)'
< HTTP/2 103
< link: </_expo/static/css/reset-f5d396.css>; as=style; rel=preload, …
< HTTP/2 200
< link: </_expo/static/css/reset-f5d396.css>; rel=preload; as=style, …

The 103 arrives first, then the 200. Cloudflare reorders the attributes in the hint, which is harmless. Don’t expect every request to get a 103. Two of my first three did. The first visit after a change only stores the header, and Cloudflare skips the 103 when the final response headers are already available.

Then go back to the analytics query above and filter on requestSource: "earlyHintsCache", grouped by status and path. On the pages you fixed, misses should give way to hits.

What changed, and the caveat

#

Minutes after the header went live, the zone logged its first Early Hints hits ever, and curl showed real 103 responses on the web pages. One header fixed both problems: the feature started working, and a slice of the “504s” went away.

The caveat is that a JSON API has nothing to preload. The API endpoints keep missing and keep writing 504 lines, and on an API-heavy zone that’s still most of the volume. You have three sensible options:

1. Send the header on the pages that benefit and accept that API paths keep logging misses. The feature works, and the charts are quieter but not silent.
2. Exclude earlyHintsCache from your 5xx alerting and dashboards. Do this whatever else you decide. An internal cache lookup should never page anyone.
3. Turn Early Hints off (Speed → Optimization in the dashboard, or the zone’s early_hints setting in the API) if you’ll never send Link headers. A single-page app whose HTML is tiny may get little from it, and then every lookup stops and the noise goes with it.

I kept it on. The web pages now get a head start, the API’s misses are filtered out of everything that alerts, and that’s what I should have done in September instead of switching the feature off.

Checklist

#

1. Before you call a Cloudflare 5xx spike an outage, split it by requestSource (or user agent).
2. earlyHintsCache 504s are cache misses with OriginResponseStatus = 0. They are harmless and never reach your origin.
3. A real 504 comes from your proxy or load balancer. Cloudflare’s own timeouts are 522 (can’t connect) and 524 (100 seconds with no response).
4. Early Hints needs the Link response header. Preload tags in your HTML don’t count.
5. Hint only render-critical CSS, scripts and fonts, and keep crossorigin on fonts.
6. Verify with curl -sv --http2: you should see a 103 before the 200.
7. Keep earlyHintsCache out of your alerting.
8. If you’ll never send the header, turn Early Hints off rather than paying for it in log noise.