<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>PgBouncer · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/pgbouncer/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Tue, 17 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/pgbouncer/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Cloudflare 520, nginx 499 and Postgres Stalls: Keeping Webhooks Alive</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</guid>
      <pubDate>Tue, 17 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>PostgreSQL</category>
      <category>PgBouncer</category>
      <category>Django</category>
      <category>Cloudflare</category>
      <category>nginx</category>
      <category>Webhooks</category>
      <category>Site Reliability</category>
      <category>Observability</category>
      <category>DevOps</category>
      <description>A 20-second Postgres stall timed out a chatbot&#39;s webhooks. How I held and retried them, and fixed Cloudflare 520 errors, nginx 499s and an exhausted PgBouncer pool.</description>
      <content:encoded><![CDATA[<p>The database behind a chatbot service I look after recently stalled twice, for about 20 seconds each time. Nothing crashed. Postgres came back by itself, the app came back with it, and the error rate barely moved. But the service&rsquo;s job is to take chat messages from a partner&rsquo;s system and acknowledge them quickly, and for those 20 seconds an acknowledgement that normally takes 0.4 s took 10 to 20 s. The partner&rsquo;s system doesn&rsquo;t wait that long. It gave up on those messages, and nothing on our side told anyone.</p>
<p>What caused them is still under investigation. The suspects so far are heavy queries, in particular ones that compare long text fields row by row, and several have been rewritten since. But none is confirmed as the cause yet.</p>
<p>So this post is about the other half of the problem, what the service does while the database isn&rsquo;t answering, and what I built after the stalls: a way to find out when the database stalls, a spool that holds requests on disk until it&rsquo;s back, a replay that runs each one exactly once, and a pass over every timeout between the internet and Postgres. A few days later it got a real test I hadn&rsquo;t planned.</p>
<p>If you landed here from one of these errors, each has its own section: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" >Cloudflare 520</a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" ><code>server closed the connection unexpectedly</code></a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#nginx-499-client-closed-request-replies-nobody-received" >nginx 499</a> and <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#pgbouncer-pool-exhausted-the-real-test" >a PgBouncer pool that ran dry</a>.</p>
<figure><div class="film-pair">
  <div class="mc-film film-dark">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRooAAABXRUJQVlA4IH4AAAAQBQCdASogABIAPw1urEqmIyIYDAQAYxDE9IAECBxAGNSFBxYLELF/yzl596pm6x2gAP7r1B4uhu6S+JqEfZ/87AtHGoj32Tp5yLTqb/5AlWmTYy/0oE8Nrih94UDfWuilEh2jC+eSMIqPA7MB5pMgkX94ddaOOys6I/gAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <div class="mc-film film-light">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light-poster.jpg" style="background:url('data:image/webp;base64,UklGRowAAABXRUJQVlA4IIAAAAAQBQCdASogABIAPw12qkmmI6GYDAQAYxDE9IAHOC3BZAFaEzG+kH6nxw8hG6nIiLJgAP7zbqngFyuj4/0DXJ2xZhiK6QUnRz4fsBZ+3lqB7T1fFerRg31UlZF+A/PCzrCTelVcbpcgNZFBnWzNRBEskzdVca60ciOypAYYRhAAAA==') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script></div><figcaption>This post as an eight-minute film, if you&rsquo;d rather watch than read. Some footage from Coverr.</figcaption>
</figure>


<h2 class="relative group">The requests
    <div id="the-requests" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-requests" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The partner&rsquo;s system sends each new chat message to a webhook endpoint. We answer <code>202 Accepted</code> with a job ID, a background worker generates the bot&rsquo;s reply, and the reply is posted back to a URL the partner gave us. Creating the job is one row in Postgres, so the 202 needs the database. Normally that takes about 0.4 s, and 2.6 s at the 99th percentile.</p>
<p>There&rsquo;s a second kind of client too: a browser extension that asks for a reply and waits for it in the same request, for up to 60 seconds.</p>
<p>The stack is ordinary. Cloudflare in front, nginx, Django, PgBouncer in transaction mode, and Postgres replicated across two machines in different places, with HAProxy sending connections to whichever one is the primary. The app server that takes this traffic runs next to the replica, so its writes cross the internet to the primary. That detail matters later.</p>

<h2 class="relative group">Detecting a Postgres stall before your users do
    <div id="detecting-a-postgres-stall-before-your-users-do" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#detecting-a-postgres-stall-before-your-users-do" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The first fix was the cheapest: know when it happens. The checks we had watched uptime, connection counts and locks, and a 20-second stall tripped none of them. Requests just took longer.</p>
<p>So I added a probe that runs on the host, outside the app, and does what the app does: it connects through the same proxy and commits a transaction, every two seconds. <code>txid_current()</code> makes Postgres assign a transaction ID, which makes it a write without touching any table. Normally it takes about 5 ms. Two commits slower than 2 s is a warning; one slower than 10 s, failing, or hanging is critical.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">psycopg</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">DSN</span> <span class="o">=</span> <span class="s2">&#34;postgresql://monitor@db-proxy:5432/app&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">commit_time</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">psycopg</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">DSN</span><span class="p">,</span> <span class="n">connect_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SET statement_timeout = &#39;15s&#39;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SELECT txid_current()&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">took</span> <span class="o">=</span> <span class="n">commit_time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database commit failed: </span><span class="si">{</span><span class="n">exc</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">took</span> <span class="o">&gt;=</span> <span class="mi">10</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database stalled: a commit took </span><span class="si">{</span><span class="n">took</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> s&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The same script reads the web server&rsquo;s access log and checks how long the 202s took in the last minute: three over 5 s is a warning, any over 10 s is critical. That&rsquo;s the number the partner actually feels.</p>
<p>Two choices in it are deliberate. It runs on the host, not inside the app, so it still works when the app is the thing that&rsquo;s stuck. And it sends alerts straight to the chat channel, not through the app&rsquo;s own alerting, for the same reason. Each condition has a 15-minute cooldown and sends a message when it clears, so a flapping database doesn&rsquo;t flood the channel.</p>

<h2 class="relative group">Holding webhook requests on disk while the database is stuck
    <div id="holding-webhook-requests-on-disk-while-the-database-is-stuck" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#holding-webhook-requests-on-disk-while-the-database-is-stuck" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Knowing about a stall doesn&rsquo;t save the requests in it. For that, the API has to answer the partner within its patience even when the database can&rsquo;t.</p>
<p>Django middleware does it. It runs the real view in a thread and waits up to a budget of 4 seconds, which is well above the normal 99th percentile. If the view finishes in time, its response goes back as usual. If it doesn&rsquo;t, the middleware writes the raw request to a spool directory on disk and answers with a 202 of its own:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span> <span class="nt">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="nt">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span> <span class="nt">&#34;held&#34;</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span></span></span></code></pre></div></div>
<p>The view&rsquo;s thread keeps running, and if the database comes back quickly it finishes the job itself. The partner&rsquo;s documentation says to treat a held 202 exactly like a normal one and not to resend it.</p>
<p>Here&rsquo;s the shape of it, cut down:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SPOOL</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s2">&#34;/var/spool/myapp/held&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">BUDGET_SECONDS</span> <span class="o">=</span> <span class="mf">4.0</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">HoldOnStallMiddleware</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">get_response</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span> <span class="o">=</span> <span class="n">get_response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">path</span> <span class="o">!=</span> <span class="s2">&#34;/api/jobs/&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">body</span>  <span class="c1"># read it before another thread can</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span><span class="o">.</span><span class="n">hex</span>
</span></span><span class="line"><span class="cl">        <span class="n">arrived</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">result</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">def</span> <span class="nf">run</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">worker</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">run</span><span class="p">,</span> <span class="n">daemon</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BUDGET_SECONDS</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="s2">&#34;response&#34;</span> <span class="ow">in</span> <span class="n">result</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">SPOOL</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">parents</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">held</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;key&#34;</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="p">,</span> <span class="s2">&#34;arrived&#34;</span><span class="p">:</span> <span class="n">arrived</span><span class="p">,</span> <span class="s2">&#34;body&#34;</span><span class="p">:</span> <span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">()}</span>
</span></span><span class="line"><span class="cl">        <span class="p">(</span><span class="n">SPOOL</span> <span class="o">/</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">arrived</span><span class="si">:</span><span class="s2">.6f</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="si">}</span><span class="s2">.json&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">write_text</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">held</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="s2">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="s2">&#34;held&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">202</span><span class="p">)</span></span></span></code></pre></div></div>
<p>It sits ahead of every middleware that touches the database, including sessions and authentication, or those would stall first and the budget would never start.</p>

<h2 class="relative group">Retrying held webhooks exactly once
    <div id="retrying-held-webhooks-exactly-once" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#retrying-held-webhooks-exactly-once" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A separate process drains the spool. It waits until <code>SELECT 1</code> answers, then replays the held requests oldest first through the real endpoint, with a signed header that carries the hold key and the original arrival time. A 202 removes the file. A 429 or a 5xx means try again later, up to six hours. Anything else moves the file to a <code>failed/</code> folder for a person to look at. A lock file keeps it to one drainer per host, and it exits every hour so the next one always runs the current code.</p>
<p>The hard part is &ldquo;exactly once&rdquo;. When the database comes back, two things may try to create the same job: the original request thread, which was still waiting, and the drainer. The obvious answer is to hand the request over in memory, so the thread stops and leaves it to the drainer. A stall test in a sandbox showed why that doesn&rsquo;t work: the stall can start after the thread has already sent its <code>INSERT</code>. The thread can&rsquo;t take back a write it&rsquo;s waiting on.</p>
<p>So the database decides. Every job stores its hold key under a partial unique index, and whichever insert comes second fails:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">IntegrityError</span><span class="p">,</span> <span class="n">models</span><span class="p">,</span> <span class="n">transaction</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Job</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">hold_key</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">blank</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">payload</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">JSONField</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">constraints</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">            <span class="n">models</span><span class="o">.</span><span class="n">UniqueConstraint</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;hold_key&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">                <span class="n">condition</span><span class="o">=</span><span class="n">models</span><span class="o">.</span><span class="n">Q</span><span class="p">(</span><span class="n">hold_key__isnull</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">                <span class="n">name</span><span class="o">=</span><span class="s2">&#34;job_hold_key_uniq&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_job</span><span class="p">(</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">with</span> <span class="n">transaction</span><span class="o">.</span><span class="n">atomic</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">IntegrityError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">)</span></span></span></code></pre></div></div>
<p>If creating a job also cancels an older one (ours does, when a newer request arrives for the same webhook), both statements go in one transaction, so the loser of the race cancels nothing.</p>
<p>Arrival time matters too. A held request is judged by when it arrived, not when it was replayed. It only replaces jobs created before it arrived, and if a newer request for the same webhook was queued while it sat on disk, it&rsquo;s dropped as superseded rather than run late.</p>

<h2 class="relative group">The first false alarm
    <div id="the-first-false-alarm" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-first-false-alarm" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Within hours of going live, the spool was holding three requests an hour, with no stall anywhere. Each was replayed within seconds, and each sent a warning and a summary to the ops channel, which buried anything real.</p>
<p>The cause was in front of the 202. For some requests the view described an image before creating the job, which took 1.6 to 2.1 s for an image it hadn&rsquo;t seen before, and occasionally pushed the acknowledgement past 4 s. The worker describes the image again anyway, so the early call only made the 202 slower. It moved into the worker, and the drainer now alerts only when a held request has waited 30 seconds without being replayed.</p>
<p>A spool with a short budget turns out to be a good way to find everything slow that sits in front of an acknowledgement.</p>

<h2 class="relative group">Finding the failing hop with Cloudflare analytics
    <div id="finding-the-failing-hop-with-cloudflare-analytics" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#finding-the-failing-hop-with-cloudflare-analytics" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Our own logs only see requests that reach the app. When something fails between the client and Django, the app has nothing to say about it. Cloudflare sees every request first, and it can tell you which side failed.</p>
<p>The zone is on the free plan, which has no Logpush, but the GraphQL Analytics API works on it. It gives request counts by edge status over time, percentiles of <code>originResponseDurationMs</code> (how long our side took to answer Cloudflare, including the proxy and the network), and sampled records of individual requests with both the status Cloudflare returned and the status our origin returned:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -s https://api.cloudflare.com/client/v4/graphql <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Authorization: Bearer </span><span class="nv">$CF_ANALYTICS_TOKEN</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  --data @- <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">{&#34;query&#34;: &#34;query($zone: String!, $since: Time!) { viewer { zones(filter: {zoneTag: $zone}) { httpRequestsAdaptive(limit: 50, orderBy: [datetime_DESC], filter: {datetime_geq: $since, edgeResponseStatus_geq: 499}) { datetime edgeResponseStatus originResponseStatus originResponseDurationMs clientRequestPath } } } }&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s"> &#34;variables&#34;: {&#34;zone&#34;: &#34;YOUR_ZONE_ID&#34;, &#34;since&#34;: &#34;YOUR_START_TIME&#34;}}
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span></span></span></code></pre></div></div>
<p><code>YOUR_START_TIME</code> is an ISO 8601 time in UTC. The Cloudflare analytics API token only needs the zone&rsquo;s Analytics Read permission. I put the results on an internal page next to the app&rsquo;s own timings.</p>
<p>Three things made it useful.</p>
<p><strong>Origin time against our own time.</strong> Cloudflare&rsquo;s origin time includes everything after the request leaves Cloudflare; the app&rsquo;s request traces cover only our code. The first reading was an origin median of 730 to 895 ms against about 500 ms of handling in the app, so a few hundred milliseconds were going somewhere before our code ran.</p>
<p><strong>The errors that never reach your logs.</strong> A 499 means the client hung up before we answered. 520 to 526 mean Cloudflare couldn&rsquo;t get a good answer from the origin. Neither shows up in Django. (A 504 from Cloudflare can be a different story again: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >some aren&rsquo;t real</a>.)</p>
<p><strong>The <code>CF-Ray</code> header.</strong> Cloudflare adds it to every request it forwards. The app now stores it on each request&rsquo;s trace, so a request Cloudflare saw can be matched to one the app handled. An edge error with no matching trace never reached the app, which narrows the problem to the hops in between.</p>
<p>That last test found two problems.</p>

<h3 class="relative group">Cloudflare 520 error fix: nginx keepalive_timeout longer than Cloudflare&rsquo;s
    <div id="cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Cloudflare returned three 520s on the webhook endpoint one day, the error its page describes as &ldquo;Web server is returning an unknown error&rdquo;, and none of them had a trace in the app. So it was the proxy.</p>
<p>Cloudflare keeps idle connections to your origin open and reuses them for up to 900 seconds. nginx was closing idle keepalive connections after 90. Now and then Cloudflare sent a request down a connection at the moment nginx closed it, and got nothing back. The fix is one line:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="c1"># Longer than Cloudflare reuses an idle connection to the origin (900 s)
</span></span></span><span class="line"><span class="cl"><span class="k">keepalive_timeout</span> <span class="s">930s</span><span class="p">;</span></span></span></code></pre></div></div>

<h3 class="relative group">server closed the connection unexpectedly (Postgres): HAProxy and PgBouncer timeouts
    <div id="server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>It was the same kind of bug I&rsquo;d fixed lower down months before. HAProxy&rsquo;s 30-second timeouts were closing PgBouncer&rsquo;s idle connections to Postgres, and the app saw <code>server closed the connection unexpectedly</code>. HAProxy now allows 600 seconds, longer than PgBouncer keeps an idle server connection (300), and Django checks a reused connection before handing it out (<code>CONN_HEALTH_CHECKS = True</code>). The rule is the same at both layers: whoever holds an idle connection open must keep it longer than whoever reuses it.</p>

<h3 class="relative group">nginx 499 (client closed request): replies nobody received
    <div id="nginx-499-client-closed-request-replies-nobody-received" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#nginx-499-client-closed-request-replies-nobody-received" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When a client hangs up, Django still finishes the request and records a 200. Only nginx knows the client was gone, and it logs a 499. Logging those separately, with the request ID the app also uses, showed one office losing about 60% of its replies in the browser extension, to page reloads in the middle of a reply.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="k">map</span> <span class="nv">$status</span> <span class="nv">$client_gone</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kn">499</span>     <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">default</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">log_format</span> <span class="s">client_aborts</span> <span class="s">escape=json</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;</span><span class="p">{</span><span class="kn">&#34;time&#34;:&#34;$time_iso8601&#34;,&#34;request_id&#34;:&#34;$request_id&#34;,&#39;</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;&#34;uri&#34;:&#34;</span><span class="nv">$uri&#34;,&#34;waited&#34;:$request_time}&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">access_log</span> <span class="s">/var/log/nginx/client-aborts.log</span> <span class="s">client_aborts</span> <span class="s">if=</span><span class="nv">$client_gone</span><span class="p">;</span></span></span></code></pre></div></div>
<p>A small job reads that file every 15 seconds and attaches each abort to its request trace once the request has finished, so a lost reply has a person and a conversation against it. An abort after 55 seconds or more is the extension&rsquo;s own 60-second timeout; anything shorter was usually a page reload.</p>
<p>One thing that doesn&rsquo;t fix a 499 is raising nginx&rsquo;s timeouts. The client is the one that gave up, so the fix is on the client&rsquo;s side or in answering sooner, and the useful part on the server is knowing which replies were lost.</p>

<h2 class="relative group">PgBouncer pool exhausted: the real test
    <div id="pgbouncer-pool-exhausted-the-real-test" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pgbouncer-pool-exhausted-the-real-test" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Then someone opened a statistics page. It sent one request per person on the team, 69 at once, and each pulled every row of that person&rsquo;s history through PgBouncer to the primary, across the internet. The pool has 50 connections plus 20 in reserve, and every one of them was taken.</p>
<p>For 1 minute 46 seconds, nothing that needed the database could get it. The spool held 80 webhook requests. All 80 were accounted for afterwards: 74 delivered, and 6 dropped because a newer request for the same webhook had replaced them. The browser extension has no spool, and 14 of its replies were lost.</p>
<p>So the spool worked, and the cause was ours. The fixes were about keeping heavy reads away from the path that serves requests:</p>
<ul>
<li><strong>One request instead of 69.</strong> The page asks for everyone at once, and the counts and averages are worked out in SQL instead of in Python over every row.</li>
<li><strong>Store what you&rsquo;ll want to average.</strong> The page scored each row by comparing two long text fields, in Python, for every row, which is why it pulled every row&rsquo;s full text across the internet. The score is now saved when the row is saved. For older rows, pairs that are identical count as a perfect score in SQL, and only the 2% or so that differ are fetched and compared. A timestamp a response-time query used to dig out of each conversation&rsquo;s history is saved when the row is created. That query used to scan up to 200 earlier messages per row; on a two-day window it went from 19.3 s to 1.5 s.</li>
<li><strong>Read from the node on the same machine.</strong> Reports get their own database connection to whichever Postgres node runs on the same host, read-only, with a statement timeout. If it&rsquo;s a replica more than five minutes behind, they use the normal connection instead.</li>
<li><strong>A cap.</strong> At most two report queries run at once per process; the rest wait up to 60 seconds and then get a 503.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># settings.py: a separate connection for heavy reads</span>
</span></span><span class="line"><span class="cl"><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;reports&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="o">**</span><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;HOST&#34;</span><span class="p">:</span> <span class="s2">&#34;localhost&#34;</span><span class="p">,</span>  <span class="c1"># the Postgres node on this machine</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;CONN_MAX_AGE&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;OPTIONS&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;connect_timeout&#34;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;options&#34;</span><span class="p">:</span> <span class="s2">&#34;-c statement_timeout=120s -c default_transaction_read_only=on&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># reports.py: at most two heavy reads per process</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">_slots</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">BoundedSemaphore</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">heavy_read</span><span class="p">(</span><span class="n">view</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="nd">@wraps</span><span class="p">(</span><span class="n">view</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">_slots</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="s2">&#34;Reports are busy. Try again shortly.&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">_slots</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">wrapper</span></span></span></code></pre></div></div>
<p>One mistake in the first version is worth repeating. A failed report read was retried once on the normal connection, which made sense for a lost connection. But it also did it for a query that had hit its statement timeout, so the slowest query on the page ran for another 120 seconds on the primary, through the shared pool. Now only a lost connection falls back, a replica conflict is retried on the same node, and a timeout is reported as it is.</p>

<h2 class="relative group">Every timeout from Cloudflare to Postgres
    <div id="every-timeout-from-cloudflare-to-postgres" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-timeout-from-cloudflare-to-postgres" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most of what went wrong was two settings that only make sense together, set in different places by different people at different times. So here they all are, from the outside in:</p>
<table>
	<thead>
			<tr>
					<th>Hop</th>
					<th>Setting</th>
					<th>Value</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Partner&rsquo;s system</td>
					<td>Waits for our 202</td>
					<td>a few seconds (the exact limit is still under investigation)</td>
			</tr>
			<tr>
					<td>Our 202</td>
					<td>Hold budget before spooling</td>
					<td>4 s (normal p99 2.6 s)</td>
			</tr>
			<tr>
					<td>Browser extension</td>
					<td>Gives up on a reply</td>
					<td>60 s</td>
			</tr>
			<tr>
					<td>Cloudflare → nginx</td>
					<td>Reuses an idle connection</td>
					<td>up to 900 s</td>
			</tr>
			<tr>
					<td>nginx</td>
					<td><code>keepalive_timeout</code></td>
					<td>930 s</td>
			</tr>
			<tr>
					<td>Cloudflare</td>
					<td>Waits for the origin&rsquo;s response</td>
					<td>100 s (then a 524, &ldquo;A timeout occurred&rdquo;)</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td><code>timeout client</code> / <code>timeout server</code></td>
					<td>600 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>server_idle_timeout</code></td>
					<td>300 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>query_wait_timeout</code> (waiting for a free connection)</td>
					<td>120 s</td>
			</tr>
			<tr>
					<td>Django</td>
					<td><code>CONN_MAX_AGE</code>, <code>CONN_HEALTH_CHECKS</code></td>
					<td>60 s, on</td>
			</tr>
			<tr>
					<td>Report connection</td>
					<td><code>statement_timeout</code>, <code>connect_timeout</code></td>
					<td>120 s, 3 s</td>
			</tr>
	</tbody>
</table>
<p>One pair in that table is still under investigation. If the report pages are served through Cloudflare&rsquo;s proxy, their 120-second statement timeout outlasts Cloudflare&rsquo;s 100 seconds, and a slow report would end in a 524 from Cloudflare before Postgres gives up on it.</p>
<p>Two rules cover nearly all of it. Idle timeouts should get longer as you go inwards: each hop keeps an idle connection open longer than the hop in front of it might reuse it. Waiting times should get shorter as you go inwards: each hop gives up, or answers something useful, before the one in front of it gives up on it. The 4-second hold budget is the second rule applied to the 202. Not every row follows it yet: PgBouncer will still queue a request for a free connection for 120 seconds, longer than either client waits.</p>

<h2 class="relative group">What I&rsquo;d do first next time
    <div id="what-id-do-first-next-time" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-do-first-next-time" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If I were setting this up again, I&rsquo;d start with the probe, because it costs an hour and tells you what you&rsquo;re dealing with. Then the Cloudflare analytics with <code>CF-Ray</code> on every trace, because it tells you which hop to look at. The spool came third, and it&rsquo;s the part I&rsquo;m happiest with, but it only made sense once I knew stalls were real and how long they lasted.</p>
<p>I&rsquo;d also take the browser extension&rsquo;s 14 lost replies as a hint. It has no spool because it waits for its answer in the same request, so there&rsquo;s nothing to hold. Giving it a job ID and a way to pick the answer up later would let it ride out the next stall the same way the webhooks did.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
