<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Site Reliability · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/site-reliability/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Tue, 17 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/site-reliability/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Cloudflare 520, nginx 499 and Postgres Stalls: Keeping Webhooks Alive</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</guid>
      <pubDate>Tue, 17 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>PostgreSQL</category>
      <category>PgBouncer</category>
      <category>Django</category>
      <category>Cloudflare</category>
      <category>nginx</category>
      <category>Webhooks</category>
      <category>Site Reliability</category>
      <category>Observability</category>
      <category>DevOps</category>
      <description>A 20-second Postgres stall timed out a chatbot&#39;s webhooks. How I held and retried them, and fixed Cloudflare 520 errors, nginx 499s and an exhausted PgBouncer pool.</description>
      <content:encoded><![CDATA[<p>The database behind a chatbot service I look after recently stalled twice, for about 20 seconds each time. Nothing crashed. Postgres came back by itself, the app came back with it, and the error rate barely moved. But the service&rsquo;s job is to take chat messages from a partner&rsquo;s system and acknowledge them quickly, and for those 20 seconds an acknowledgement that normally takes 0.4 s took 10 to 20 s. The partner&rsquo;s system doesn&rsquo;t wait that long. It gave up on those messages, and nothing on our side told anyone.</p>
<p>What caused them is still under investigation. The suspects so far are heavy queries, in particular ones that compare long text fields row by row, and several have been rewritten since. But none is confirmed as the cause yet.</p>
<p>So this post is about the other half of the problem, what the service does while the database isn&rsquo;t answering, and what I built after the stalls: a way to find out when the database stalls, a spool that holds requests on disk until it&rsquo;s back, a replay that runs each one exactly once, and a pass over every timeout between the internet and Postgres. A few days later it got a real test I hadn&rsquo;t planned.</p>
<p>If you landed here from one of these errors, each has its own section: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" >Cloudflare 520</a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" ><code>server closed the connection unexpectedly</code></a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#nginx-499-client-closed-request-replies-nobody-received" >nginx 499</a> and <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#pgbouncer-pool-exhausted-the-real-test" >a PgBouncer pool that ran dry</a>.</p>
<figure><div class="film-pair">
  <div class="mc-film film-dark">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRooAAABXRUJQVlA4IH4AAAAQBQCdASogABIAPw1urEqmIyIYDAQAYxDE9IAECBxAGNSFBxYLELF/yzl596pm6x2gAP7r1B4uhu6S+JqEfZ/87AtHGoj32Tp5yLTqb/5AlWmTYy/0oE8Nrih94UDfWuilEh2jC+eSMIqPA7MB5pMgkX94ddaOOys6I/gAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <div class="mc-film film-light">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light-poster.jpg" style="background:url('data:image/webp;base64,UklGRowAAABXRUJQVlA4IIAAAAAQBQCdASogABIAPw12qkmmI6GYDAQAYxDE9IAHOC3BZAFaEzG+kH6nxw8hG6nIiLJgAP7zbqngFyuj4/0DXJ2xZhiK6QUnRz4fsBZ+3lqB7T1fFerRg31UlZF+A/PCzrCTelVcbpcgNZFBnWzNRBEskzdVca60ciOypAYYRhAAAA==') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script></div><figcaption>This post as an eight-minute film, if you&rsquo;d rather watch than read. Some footage from Coverr.</figcaption>
</figure>


<h2 class="relative group">The requests
    <div id="the-requests" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-requests" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The partner&rsquo;s system sends each new chat message to a webhook endpoint. We answer <code>202 Accepted</code> with a job ID, a background worker generates the bot&rsquo;s reply, and the reply is posted back to a URL the partner gave us. Creating the job is one row in Postgres, so the 202 needs the database. Normally that takes about 0.4 s, and 2.6 s at the 99th percentile.</p>
<p>There&rsquo;s a second kind of client too: a browser extension that asks for a reply and waits for it in the same request, for up to 60 seconds.</p>
<p>The stack is ordinary. Cloudflare in front, nginx, Django, PgBouncer in transaction mode, and Postgres replicated across two machines in different places, with HAProxy sending connections to whichever one is the primary. The app server that takes this traffic runs next to the replica, so its writes cross the internet to the primary. That detail matters later.</p>

<h2 class="relative group">Detecting a Postgres stall before your users do
    <div id="detecting-a-postgres-stall-before-your-users-do" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#detecting-a-postgres-stall-before-your-users-do" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The first fix was the cheapest: know when it happens. The checks we had watched uptime, connection counts and locks, and a 20-second stall tripped none of them. Requests just took longer.</p>
<p>So I added a probe that runs on the host, outside the app, and does what the app does: it connects through the same proxy and commits a transaction, every two seconds. <code>txid_current()</code> makes Postgres assign a transaction ID, which makes it a write without touching any table. Normally it takes about 5 ms. Two commits slower than 2 s is a warning; one slower than 10 s, failing, or hanging is critical.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">psycopg</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">DSN</span> <span class="o">=</span> <span class="s2">&#34;postgresql://monitor@db-proxy:5432/app&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">commit_time</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">psycopg</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">DSN</span><span class="p">,</span> <span class="n">connect_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SET statement_timeout = &#39;15s&#39;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SELECT txid_current()&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">took</span> <span class="o">=</span> <span class="n">commit_time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database commit failed: </span><span class="si">{</span><span class="n">exc</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">took</span> <span class="o">&gt;=</span> <span class="mi">10</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database stalled: a commit took </span><span class="si">{</span><span class="n">took</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> s&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The same script reads the web server&rsquo;s access log and checks how long the 202s took in the last minute: three over 5 s is a warning, any over 10 s is critical. That&rsquo;s the number the partner actually feels.</p>
<p>Two choices in it are deliberate. It runs on the host, not inside the app, so it still works when the app is the thing that&rsquo;s stuck. And it sends alerts straight to the chat channel, not through the app&rsquo;s own alerting, for the same reason. Each condition has a 15-minute cooldown and sends a message when it clears, so a flapping database doesn&rsquo;t flood the channel.</p>

<h2 class="relative group">Holding webhook requests on disk while the database is stuck
    <div id="holding-webhook-requests-on-disk-while-the-database-is-stuck" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#holding-webhook-requests-on-disk-while-the-database-is-stuck" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Knowing about a stall doesn&rsquo;t save the requests in it. For that, the API has to answer the partner within its patience even when the database can&rsquo;t.</p>
<p>Django middleware does it. It runs the real view in a thread and waits up to a budget of 4 seconds, which is well above the normal 99th percentile. If the view finishes in time, its response goes back as usual. If it doesn&rsquo;t, the middleware writes the raw request to a spool directory on disk and answers with a 202 of its own:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span> <span class="nt">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="nt">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span> <span class="nt">&#34;held&#34;</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span></span></span></code></pre></div></div>
<p>The view&rsquo;s thread keeps running, and if the database comes back quickly it finishes the job itself. The partner&rsquo;s documentation says to treat a held 202 exactly like a normal one and not to resend it.</p>
<p>Here&rsquo;s the shape of it, cut down:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SPOOL</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s2">&#34;/var/spool/myapp/held&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">BUDGET_SECONDS</span> <span class="o">=</span> <span class="mf">4.0</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">HoldOnStallMiddleware</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">get_response</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span> <span class="o">=</span> <span class="n">get_response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">path</span> <span class="o">!=</span> <span class="s2">&#34;/api/jobs/&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">body</span>  <span class="c1"># read it before another thread can</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span><span class="o">.</span><span class="n">hex</span>
</span></span><span class="line"><span class="cl">        <span class="n">arrived</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">result</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">def</span> <span class="nf">run</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">worker</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">run</span><span class="p">,</span> <span class="n">daemon</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BUDGET_SECONDS</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="s2">&#34;response&#34;</span> <span class="ow">in</span> <span class="n">result</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">SPOOL</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">parents</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">held</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;key&#34;</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="p">,</span> <span class="s2">&#34;arrived&#34;</span><span class="p">:</span> <span class="n">arrived</span><span class="p">,</span> <span class="s2">&#34;body&#34;</span><span class="p">:</span> <span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">()}</span>
</span></span><span class="line"><span class="cl">        <span class="p">(</span><span class="n">SPOOL</span> <span class="o">/</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">arrived</span><span class="si">:</span><span class="s2">.6f</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="si">}</span><span class="s2">.json&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">write_text</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">held</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="s2">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="s2">&#34;held&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">202</span><span class="p">)</span></span></span></code></pre></div></div>
<p>It sits ahead of every middleware that touches the database, including sessions and authentication, or those would stall first and the budget would never start.</p>

<h2 class="relative group">Retrying held webhooks exactly once
    <div id="retrying-held-webhooks-exactly-once" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#retrying-held-webhooks-exactly-once" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A separate process drains the spool. It waits until <code>SELECT 1</code> answers, then replays the held requests oldest first through the real endpoint, with a signed header that carries the hold key and the original arrival time. A 202 removes the file. A 429 or a 5xx means try again later, up to six hours. Anything else moves the file to a <code>failed/</code> folder for a person to look at. A lock file keeps it to one drainer per host, and it exits every hour so the next one always runs the current code.</p>
<p>The hard part is &ldquo;exactly once&rdquo;. When the database comes back, two things may try to create the same job: the original request thread, which was still waiting, and the drainer. The obvious answer is to hand the request over in memory, so the thread stops and leaves it to the drainer. A stall test in a sandbox showed why that doesn&rsquo;t work: the stall can start after the thread has already sent its <code>INSERT</code>. The thread can&rsquo;t take back a write it&rsquo;s waiting on.</p>
<p>So the database decides. Every job stores its hold key under a partial unique index, and whichever insert comes second fails:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">IntegrityError</span><span class="p">,</span> <span class="n">models</span><span class="p">,</span> <span class="n">transaction</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Job</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">hold_key</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">blank</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">payload</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">JSONField</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">constraints</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">            <span class="n">models</span><span class="o">.</span><span class="n">UniqueConstraint</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;hold_key&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">                <span class="n">condition</span><span class="o">=</span><span class="n">models</span><span class="o">.</span><span class="n">Q</span><span class="p">(</span><span class="n">hold_key__isnull</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">                <span class="n">name</span><span class="o">=</span><span class="s2">&#34;job_hold_key_uniq&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_job</span><span class="p">(</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">with</span> <span class="n">transaction</span><span class="o">.</span><span class="n">atomic</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">IntegrityError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">)</span></span></span></code></pre></div></div>
<p>If creating a job also cancels an older one (ours does, when a newer request arrives for the same webhook), both statements go in one transaction, so the loser of the race cancels nothing.</p>
<p>Arrival time matters too. A held request is judged by when it arrived, not when it was replayed. It only replaces jobs created before it arrived, and if a newer request for the same webhook was queued while it sat on disk, it&rsquo;s dropped as superseded rather than run late.</p>

<h2 class="relative group">The first false alarm
    <div id="the-first-false-alarm" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-first-false-alarm" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Within hours of going live, the spool was holding three requests an hour, with no stall anywhere. Each was replayed within seconds, and each sent a warning and a summary to the ops channel, which buried anything real.</p>
<p>The cause was in front of the 202. For some requests the view described an image before creating the job, which took 1.6 to 2.1 s for an image it hadn&rsquo;t seen before, and occasionally pushed the acknowledgement past 4 s. The worker describes the image again anyway, so the early call only made the 202 slower. It moved into the worker, and the drainer now alerts only when a held request has waited 30 seconds without being replayed.</p>
<p>A spool with a short budget turns out to be a good way to find everything slow that sits in front of an acknowledgement.</p>

<h2 class="relative group">Finding the failing hop with Cloudflare analytics
    <div id="finding-the-failing-hop-with-cloudflare-analytics" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#finding-the-failing-hop-with-cloudflare-analytics" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Our own logs only see requests that reach the app. When something fails between the client and Django, the app has nothing to say about it. Cloudflare sees every request first, and it can tell you which side failed.</p>
<p>The zone is on the free plan, which has no Logpush, but the GraphQL Analytics API works on it. It gives request counts by edge status over time, percentiles of <code>originResponseDurationMs</code> (how long our side took to answer Cloudflare, including the proxy and the network), and sampled records of individual requests with both the status Cloudflare returned and the status our origin returned:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -s https://api.cloudflare.com/client/v4/graphql <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Authorization: Bearer </span><span class="nv">$CF_ANALYTICS_TOKEN</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  --data @- <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">{&#34;query&#34;: &#34;query($zone: String!, $since: Time!) { viewer { zones(filter: {zoneTag: $zone}) { httpRequestsAdaptive(limit: 50, orderBy: [datetime_DESC], filter: {datetime_geq: $since, edgeResponseStatus_geq: 499}) { datetime edgeResponseStatus originResponseStatus originResponseDurationMs clientRequestPath } } } }&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s"> &#34;variables&#34;: {&#34;zone&#34;: &#34;YOUR_ZONE_ID&#34;, &#34;since&#34;: &#34;YOUR_START_TIME&#34;}}
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span></span></span></code></pre></div></div>
<p><code>YOUR_START_TIME</code> is an ISO 8601 time in UTC. The Cloudflare analytics API token only needs the zone&rsquo;s Analytics Read permission. I put the results on an internal page next to the app&rsquo;s own timings.</p>
<p>Three things made it useful.</p>
<p><strong>Origin time against our own time.</strong> Cloudflare&rsquo;s origin time includes everything after the request leaves Cloudflare; the app&rsquo;s request traces cover only our code. The first reading was an origin median of 730 to 895 ms against about 500 ms of handling in the app, so a few hundred milliseconds were going somewhere before our code ran.</p>
<p><strong>The errors that never reach your logs.</strong> A 499 means the client hung up before we answered. 520 to 526 mean Cloudflare couldn&rsquo;t get a good answer from the origin. Neither shows up in Django. (A 504 from Cloudflare can be a different story again: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >some aren&rsquo;t real</a>.)</p>
<p><strong>The <code>CF-Ray</code> header.</strong> Cloudflare adds it to every request it forwards. The app now stores it on each request&rsquo;s trace, so a request Cloudflare saw can be matched to one the app handled. An edge error with no matching trace never reached the app, which narrows the problem to the hops in between.</p>
<p>That last test found two problems.</p>

<h3 class="relative group">Cloudflare 520 error fix: nginx keepalive_timeout longer than Cloudflare&rsquo;s
    <div id="cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Cloudflare returned three 520s on the webhook endpoint one day, the error its page describes as &ldquo;Web server is returning an unknown error&rdquo;, and none of them had a trace in the app. So it was the proxy.</p>
<p>Cloudflare keeps idle connections to your origin open and reuses them for up to 900 seconds. nginx was closing idle keepalive connections after 90. Now and then Cloudflare sent a request down a connection at the moment nginx closed it, and got nothing back. The fix is one line:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="c1"># Longer than Cloudflare reuses an idle connection to the origin (900 s)
</span></span></span><span class="line"><span class="cl"><span class="k">keepalive_timeout</span> <span class="s">930s</span><span class="p">;</span></span></span></code></pre></div></div>

<h3 class="relative group">server closed the connection unexpectedly (Postgres): HAProxy and PgBouncer timeouts
    <div id="server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>It was the same kind of bug I&rsquo;d fixed lower down months before. HAProxy&rsquo;s 30-second timeouts were closing PgBouncer&rsquo;s idle connections to Postgres, and the app saw <code>server closed the connection unexpectedly</code>. HAProxy now allows 600 seconds, longer than PgBouncer keeps an idle server connection (300), and Django checks a reused connection before handing it out (<code>CONN_HEALTH_CHECKS = True</code>). The rule is the same at both layers: whoever holds an idle connection open must keep it longer than whoever reuses it.</p>

<h3 class="relative group">nginx 499 (client closed request): replies nobody received
    <div id="nginx-499-client-closed-request-replies-nobody-received" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#nginx-499-client-closed-request-replies-nobody-received" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When a client hangs up, Django still finishes the request and records a 200. Only nginx knows the client was gone, and it logs a 499. Logging those separately, with the request ID the app also uses, showed one office losing about 60% of its replies in the browser extension, to page reloads in the middle of a reply.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="k">map</span> <span class="nv">$status</span> <span class="nv">$client_gone</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kn">499</span>     <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">default</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">log_format</span> <span class="s">client_aborts</span> <span class="s">escape=json</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;</span><span class="p">{</span><span class="kn">&#34;time&#34;:&#34;$time_iso8601&#34;,&#34;request_id&#34;:&#34;$request_id&#34;,&#39;</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;&#34;uri&#34;:&#34;</span><span class="nv">$uri&#34;,&#34;waited&#34;:$request_time}&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">access_log</span> <span class="s">/var/log/nginx/client-aborts.log</span> <span class="s">client_aborts</span> <span class="s">if=</span><span class="nv">$client_gone</span><span class="p">;</span></span></span></code></pre></div></div>
<p>A small job reads that file every 15 seconds and attaches each abort to its request trace once the request has finished, so a lost reply has a person and a conversation against it. An abort after 55 seconds or more is the extension&rsquo;s own 60-second timeout; anything shorter was usually a page reload.</p>
<p>One thing that doesn&rsquo;t fix a 499 is raising nginx&rsquo;s timeouts. The client is the one that gave up, so the fix is on the client&rsquo;s side or in answering sooner, and the useful part on the server is knowing which replies were lost.</p>

<h2 class="relative group">PgBouncer pool exhausted: the real test
    <div id="pgbouncer-pool-exhausted-the-real-test" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pgbouncer-pool-exhausted-the-real-test" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Then someone opened a statistics page. It sent one request per person on the team, 69 at once, and each pulled every row of that person&rsquo;s history through PgBouncer to the primary, across the internet. The pool has 50 connections plus 20 in reserve, and every one of them was taken.</p>
<p>For 1 minute 46 seconds, nothing that needed the database could get it. The spool held 80 webhook requests. All 80 were accounted for afterwards: 74 delivered, and 6 dropped because a newer request for the same webhook had replaced them. The browser extension has no spool, and 14 of its replies were lost.</p>
<p>So the spool worked, and the cause was ours. The fixes were about keeping heavy reads away from the path that serves requests:</p>
<ul>
<li><strong>One request instead of 69.</strong> The page asks for everyone at once, and the counts and averages are worked out in SQL instead of in Python over every row.</li>
<li><strong>Store what you&rsquo;ll want to average.</strong> The page scored each row by comparing two long text fields, in Python, for every row, which is why it pulled every row&rsquo;s full text across the internet. The score is now saved when the row is saved. For older rows, pairs that are identical count as a perfect score in SQL, and only the 2% or so that differ are fetched and compared. A timestamp a response-time query used to dig out of each conversation&rsquo;s history is saved when the row is created. That query used to scan up to 200 earlier messages per row; on a two-day window it went from 19.3 s to 1.5 s.</li>
<li><strong>Read from the node on the same machine.</strong> Reports get their own database connection to whichever Postgres node runs on the same host, read-only, with a statement timeout. If it&rsquo;s a replica more than five minutes behind, they use the normal connection instead.</li>
<li><strong>A cap.</strong> At most two report queries run at once per process; the rest wait up to 60 seconds and then get a 503.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># settings.py: a separate connection for heavy reads</span>
</span></span><span class="line"><span class="cl"><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;reports&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="o">**</span><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;HOST&#34;</span><span class="p">:</span> <span class="s2">&#34;localhost&#34;</span><span class="p">,</span>  <span class="c1"># the Postgres node on this machine</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;CONN_MAX_AGE&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;OPTIONS&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;connect_timeout&#34;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;options&#34;</span><span class="p">:</span> <span class="s2">&#34;-c statement_timeout=120s -c default_transaction_read_only=on&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># reports.py: at most two heavy reads per process</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">_slots</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">BoundedSemaphore</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">heavy_read</span><span class="p">(</span><span class="n">view</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="nd">@wraps</span><span class="p">(</span><span class="n">view</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">_slots</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="s2">&#34;Reports are busy. Try again shortly.&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">_slots</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">wrapper</span></span></span></code></pre></div></div>
<p>One mistake in the first version is worth repeating. A failed report read was retried once on the normal connection, which made sense for a lost connection. But it also did it for a query that had hit its statement timeout, so the slowest query on the page ran for another 120 seconds on the primary, through the shared pool. Now only a lost connection falls back, a replica conflict is retried on the same node, and a timeout is reported as it is.</p>

<h2 class="relative group">Every timeout from Cloudflare to Postgres
    <div id="every-timeout-from-cloudflare-to-postgres" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-timeout-from-cloudflare-to-postgres" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most of what went wrong was two settings that only make sense together, set in different places by different people at different times. So here they all are, from the outside in:</p>
<table>
	<thead>
			<tr>
					<th>Hop</th>
					<th>Setting</th>
					<th>Value</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Partner&rsquo;s system</td>
					<td>Waits for our 202</td>
					<td>a few seconds (the exact limit is still under investigation)</td>
			</tr>
			<tr>
					<td>Our 202</td>
					<td>Hold budget before spooling</td>
					<td>4 s (normal p99 2.6 s)</td>
			</tr>
			<tr>
					<td>Browser extension</td>
					<td>Gives up on a reply</td>
					<td>60 s</td>
			</tr>
			<tr>
					<td>Cloudflare → nginx</td>
					<td>Reuses an idle connection</td>
					<td>up to 900 s</td>
			</tr>
			<tr>
					<td>nginx</td>
					<td><code>keepalive_timeout</code></td>
					<td>930 s</td>
			</tr>
			<tr>
					<td>Cloudflare</td>
					<td>Waits for the origin&rsquo;s response</td>
					<td>100 s (then a 524, &ldquo;A timeout occurred&rdquo;)</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td><code>timeout client</code> / <code>timeout server</code></td>
					<td>600 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>server_idle_timeout</code></td>
					<td>300 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>query_wait_timeout</code> (waiting for a free connection)</td>
					<td>120 s</td>
			</tr>
			<tr>
					<td>Django</td>
					<td><code>CONN_MAX_AGE</code>, <code>CONN_HEALTH_CHECKS</code></td>
					<td>60 s, on</td>
			</tr>
			<tr>
					<td>Report connection</td>
					<td><code>statement_timeout</code>, <code>connect_timeout</code></td>
					<td>120 s, 3 s</td>
			</tr>
	</tbody>
</table>
<p>One pair in that table is still under investigation. If the report pages are served through Cloudflare&rsquo;s proxy, their 120-second statement timeout outlasts Cloudflare&rsquo;s 100 seconds, and a slow report would end in a 524 from Cloudflare before Postgres gives up on it.</p>
<p>Two rules cover nearly all of it. Idle timeouts should get longer as you go inwards: each hop keeps an idle connection open longer than the hop in front of it might reuse it. Waiting times should get shorter as you go inwards: each hop gives up, or answers something useful, before the one in front of it gives up on it. The 4-second hold budget is the second rule applied to the 202. Not every row follows it yet: PgBouncer will still queue a request for a free connection for 120 seconds, longer than either client waits.</p>

<h2 class="relative group">What I&rsquo;d do first next time
    <div id="what-id-do-first-next-time" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-do-first-next-time" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If I were setting this up again, I&rsquo;d start with the probe, because it costs an hour and tells you what you&rsquo;re dealing with. Then the Cloudflare analytics with <code>CF-Ray</code> on every trace, because it tells you which hop to look at. The spool came third, and it&rsquo;s the part I&rsquo;m happiest with, but it only made sense once I knew stalls were real and how long they lasted.</p>
<p>I&rsquo;d also take the browser extension&rsquo;s 14 lost replies as a hint. It has no spool because it waits for its answer in the same request, so there&rsquo;s nothing to hold. Giving it a job ID and a way to pick the answer up later would let it ride out the next stall the same way the webhooks did.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Monitoring Django on Fly.io: The DevOps Tools I Kept and Dropped</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/</guid>
      <pubDate>Thu, 22 Oct 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>DevOps</category>
      <category>Monitoring</category>
      <category>Observability</category>
      <category>Site Reliability</category>
      <category>Fly.io</category>
      <category>Django</category>
      <category>Cloudflare</category>
      <category>Sentry</category>
      <category>Crash Reporting</category>
      <category>Small Teams</category>
      <category>Tech Stack</category>
      <description>Monitoring two small Django apps on Fly.io for about US$2 a month: errors to chat, logs in S3 via DuckDB, Fly&#39;s metrics, crash reports, and what I dropped.</description>
      <content:encoded><![CDATA[<p>For two small products, a marketplace and community app I build and an estimating app for tradespeople, both on Fly.io&rsquo;s cheapest machines, my DevOps monitoring comes down to five things: ERROR logs posted to a chat channel, every log line archived in object storage and queried with DuckDB, Fly&rsquo;s built-in Prometheus metrics, phone crash reports sent through Sentry&rsquo;s open-source SDK to my own API, and one summary post each morning.</p>
<p>There&rsquo;s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here&rsquo;s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.</p>

<h2 class="relative group">What I mean by monitoring, and how small this is
    <div id="what-i-mean-by-monitoring-and-how-small-this-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-mean-by-monitoring-and-how-small-this-is" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>By DevOps monitoring I mean finding out that something&rsquo;s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.</p>
<p>Last year I wrote about <a href="https://jared.lynskey.co.nz/en/posts/2025/2025-10-13-devops-monitoring/" >setting up monitoring that developers actually use</a>, mostly with teams in mind: a cloud provider&rsquo;s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly&rsquo;s shared-cpu-1x machines, as described in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/" >My Current Stack</a>, and I&rsquo;m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly&rsquo;s metrics in September had the marketplace&rsquo;s API averaging 1% CPU. This isn&rsquo;t an enterprise stack, and I won&rsquo;t dress it up as one.</p>
<p>Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy&rsquo;s, on <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-01-cloudflare-warp-dev-network/" >my workstation</a>.</p>

<h2 class="relative group">The list
    <div id="the-list" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-list" aria-label="Anchor">#</a>
    </span>
    
</h2>
<table>
	<thead>
			<tr>
					<th>Tool</th>
					<th>What it watches</th>
					<th>Kept or dropped, and why</th>
					<th>Cost</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A team chat channel, fed by a Django logging handler</td>
					<td>ERRORs from the API, WebSocket server and worker; deploys; bursts of wrong sign-in codes</td>
					<td>Kept. It&rsquo;s the one place I look without being asked</td>
					<td>Free</td>
			</tr>
			<tr>
					<td>Fly&rsquo;s log shipper to an S3-compatible bucket, queried with DuckDB</td>
					<td>Every log line from the marketplace&rsquo;s apps</td>
					<td>Kept. Fly&rsquo;s own buffer holds minutes; this holds everything, in SQL</td>
					<td>About US$2 a month, plus storage</td>
			</tr>
			<tr>
					<td>Fly&rsquo;s built-in metrics (Prometheus API)</td>
					<td>CPU, memory, burst credit, responses; the managed Postgres exporter</td>
					<td>Kept. It showed the database stalls weren&rsquo;t my code</td>
					<td>Comes with Fly</td>
			</tr>
			<tr>
					<td>Fly health checks on <code>/api/health/</code> (both apps)</td>
					<td>Whether a machine should get traffic</td>
					<td>Kept for routing. Dropped as proof that users are fine</td>
					<td>Comes with Fly</td>
			</tr>
			<tr>
					<td>Sentry&rsquo;s SDK, its DSN pointed at my own API</td>
					<td>App crashes on phones</td>
					<td>Kept the SDK. Never signed up for sentry.io</td>
					<td>Nothing on top of the API</td>
			</tr>
			<tr>
					<td>A morning summary in the team chat</td>
					<td>Yesterday&rsquo;s errors, usage, search, chats and board</td>
					<td>Kept. Moved off a CI cron job onto my Mac</td>
					<td>Free</td>
			</tr>
			<tr>
					<td>An app analytics SDK (both apps), and a search engine&rsquo;s webmaster console</td>
					<td>Usage on web, iOS and Android; how search sees the site</td>
					<td>Kept, read through their APIs</td>
					<td>Free</td>
			</tr>
			<tr>
					<td>Cloudflare&rsquo;s GraphQL analytics</td>
					<td>Who is behind a status code or a path</td>
					<td>Kept, grouped by user agent before I believe it</td>
					<td>Free plan, one day per query</td>
			</tr>
			<tr>
					<td>A hosted over-the-air update service&rsquo;s dashboard</td>
					<td>Which over-the-air release phones run; crash rollbacks</td>
					<td>Moved to it from an admin list that couldn&rsquo;t tell me</td>
					<td>Part of the update service</td>
			</tr>
			<tr>
					<td>A small probe script</td>
					<td>Failed requests during a deploy or restart</td>
					<td>Kept. It hits the origin, so a cache can&rsquo;t hide a gap</td>
					<td>Free</td>
			</tr>
			<tr>
					<td><code>flyctl logs</code></td>
					<td>The last few minutes of an app&rsquo;s output</td>
					<td>Kept as a live tail. Dropped as the record</td>
					<td>Comes with Fly</td>
			</tr>
			<tr>
					<td>Cloudflare Early Hints</td>
					<td>Nothing; it&rsquo;s a speed feature</td>
					<td>Kept, once the origin sent a <code>Link</code> header. Its cache misses were every 504 on the zone</td>
					<td>Free</td>
			</tr>
	</tbody>
</table>
<p>Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly&rsquo;s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)</p>

<h2 class="relative group">Errors: a logging handler posting to chat
    <div id="errors-a-logging-handler-posting-to-chat" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#errors-a-logging-handler-posting-to-chat" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Until September, <code>flyctl logs</code> was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don&rsquo;t alert.</p>
<p>The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can&rsquo;t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand.</p>
<p>The channel also carries what I want to read errors against: deploys post when they start, when they&rsquo;re up and where they failed, with the commit and who ran it, and so do over-the-air updates and <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-14-web-releases-without-redeploying/" >web releases</a>. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.</p>
<p>In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process&rsquo;s start command, and writes the same line to that process&rsquo;s log, so an id I can&rsquo;t find means a process that doesn&rsquo;t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.</p>

<h2 class="relative group">Logs: Fly&rsquo;s shipper to S3, queried with DuckDB
    <div id="logs-flys-shipper-to-s3-queried-with-duckdb" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#logs-flys-shipper-to-s3-queried-with-duckdb" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly&rsquo;s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly&rsquo;s log shipper, which is Vector subscribed to the organisation&rsquo;s log stream, with one change to its S3 sink:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">sinks</span><span class="p">.</span><span class="nx">aws_s3</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s2">&#34;aws_s3&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">bucket</span> <span class="p">=</span> <span class="s2">&#34;myapp-logs&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">compression</span> <span class="p">=</span> <span class="s2">&#34;gzip&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">key_prefix</span> <span class="p">=</span> <span class="s2">&#34;logs/{{fly.app.name}}/%F/&#34;</span>   <span class="c"># one folder per app per day</span>
</span></span><span class="line"><span class="cl">  <span class="nx">batch</span><span class="p">.</span><span class="nx">timeout_secs</span> <span class="p">=</span> <span class="mi">300</span>                  <span class="c"># a quiet app still lands its lines</span></span></span></code></pre></div></div>
<p>Every line from the marketplace&rsquo;s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="n">MACRO</span><span class="w"> </span><span class="n">logs</span><span class="p">(</span><span class="n">app</span><span class="p">,</span><span class="w"> </span><span class="k">day</span><span class="w"> </span><span class="p">:</span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;*&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">TABLE</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">read_json_auto</span><span class="p">(</span><span class="s1">&#39;s3://myapp-logs/logs/&#39;</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="s1">&#39;/&#39;</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="k">day</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="s1">&#39;/*.log.gz&#39;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                               </span><span class="n">union_by_name</span><span class="w"> </span><span class="p">:</span><span class="o">=</span><span class="w"> </span><span class="k">true</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">timestamp</span><span class="p">,</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="p">(</span><span class="s1">&#39;myapp-api&#39;</span><span class="p">,</span><span class="w"> </span><span class="s1">&#39;2026-09-06&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">LIKE</span><span class="w"> </span><span class="s1">&#39;% 500 %&#39;</span><span class="p">;</span></span></span></code></pre></div></div>
<p>That macro is how I measured the cold starts in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >the scale-to-zero post</a>: the startup script announces each phase, every line carries the machine&rsquo;s id, and the rest is a <code>GROUP BY</code>.</p>
<p>So it&rsquo;s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.</p>

<h2 class="relative group">Metrics: Fly&rsquo;s built-in Prometheus
    <div id="metrics-flys-built-in-prometheus" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#metrics-flys-built-in-prometheus" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly runs a managed Prometheus for each organisation, so I don&rsquo;t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (<code>fly_instance_cpu_balance</code> matters on a shared CPU), responses and concurrency per app. Two things I wish I&rsquo;d known first. A token from <code>fly tokens create</code> goes in as <code>Authorization: FlyV1 &lt;token&gt;</code>; <code>Bearer</code> only works with the personal token from <code>flyctl auth token</code>, and a read-only token gets &ldquo;something went wrong resolving organization&rdquo;. And it keeps about two weeks (Fly says roughly 15 days).</p>
<p>It earned its place in September, when the logs had &ldquo;Could not open a database connection&rdquo; several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary&rsquo;s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres&rsquo;s start time never moved. Postgres hadn&rsquo;t restarted; the host under it was stalling. I sent Fly&rsquo;s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.</p>

<h2 class="relative group">Crash reports: Sentry&rsquo;s SDK, pointed at my own API
    <div id="crash-reports-sentrys-sdk-pointed-at-my-own-api" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#crash-reports-sentrys-sdk-pointed-at-my-own-api" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry&rsquo;s open-source SDK, but its DSN points at an endpoint in my own Django API that accepts Sentry&rsquo;s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It&rsquo;s errors only.</p>
<p>I didn&rsquo;t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven&rsquo;t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.</p>

<h2 class="relative group">The only dashboard is a message
    <div id="the-only-dashboard-is-a-message" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-only-dashboard-is-a-message" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I haven&rsquo;t built a dashboard for either app. Last year&rsquo;s post had a test for them: if nobody opens one unprompted, it isn&rsquo;t earning its place. A message that arrives on its own doesn&rsquo;t need opening, so the closest thing I have is a post in the team chat every morning.</p>
<p>It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can&rsquo;t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel.</p>
<p>Then failed health checks, yesterday&rsquo;s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what I&rsquo;ve spent. A section that can&rsquo;t load says so, and the post still goes out.</p>
<p>Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month&rsquo;s Android totals with them taken out.</p>
<p>Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an &ldquo;is active&rdquo; checkbox, and I had no idea how many phones had taken one. The hosted service&rsquo;s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-13-moving-to-codemagic-patch/" >the move to a hosted update service</a> separately.</p>

<h2 class="relative group">What I dropped, or stopped believing
    <div id="what-i-dropped-or-stopped-believing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-dropped-or-stopped-believing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>None of this was a paid tool. Mostly it was things I&rsquo;d been treating as signals that weren&rsquo;t, and one job running in the wrong place.</p>

<h3 class="relative group">Cloudflare&rsquo;s 504 rate, until I grouped it by user agent
    <div id="cloudflares-504-rate-until-i-grouped-it-by-user-agent" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudflares-504-rate-until-i-grouped-it-by-user-agent" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>In September the Cloudflare dashboard showed what looked like 22% of a day&rsquo;s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app&rsquo;s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >the scale-to-zero post</a> as the cause.</p>
<p>The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare&rsquo;s own Early Hints cache lookups (user agents <code>bastion early hints</code> and <code>nginx-ssl early hints</code>), which never reach the origin: a 504 on one of those lines is a cache miss, not a timeout. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn&rsquo;t doing anything for me because the origin sent no <code>Link</code> header, so I turned it off, and later <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >made it work</a>.</p>
<p>The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user.</p>
<p>So this is the rule I&rsquo;d most like to pass on: before you believe a CDN&rsquo;s error rate, group it by user agent and check it isn&rsquo;t the CDN&rsquo;s own machinery. On Cloudflare&rsquo;s free plan that&rsquo;s a GraphQL query on <code>httpRequestsAdaptiveGroups</code>, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn&rsquo;t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.</p>

<h3 class="relative group">A green health check
    <div id="a-green-health-check" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-green-health-check" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Fly checks <code>/api/health/</code> on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it&rsquo;s good at. But a health check runs against a machine that&rsquo;s already up, so it can&rsquo;t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now.</p>
<p>Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain <code>flyctl deploy</code> leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.</p>
<p>The health check has also caused two outages of its own. In July, Django&rsquo;s <code>SECURE_SSL_REDIRECT</code> answered Fly&rsquo;s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an <code>except Exception</code> in the health view let a &ldquo;connection already closed&rdquo; error escape as a 500, and Fly pulled the machine in exactly the case the view&rsquo;s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.</p>
<p>The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.</p>

<h3 class="relative group">A scheduled job in CI
    <div id="a-scheduled-job-in-ci" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-scheduled-job-in-ci" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project&rsquo;s service-account key, which can do anything in that project, and I didn&rsquo;t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it&rsquo;s past eight in Auckland and today&rsquo;s post hasn&rsquo;t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.</p>

<h2 class="relative group">If I were starting again
    <div id="if-i-were-starting-again" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#if-i-were-starting-again" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I&rsquo;d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That&rsquo;s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I&rsquo;d stop checking them.</p>
<p>I&rsquo;d leave tracing until there&rsquo;s a second service for a request to cross, and a hosted APM until there&rsquo;s a team that needs shared dashboards, an on-call rota and a budget for someone else&rsquo;s bill. That&rsquo;s when the tools in last year&rsquo;s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Monitoring for Developers: Metrics, Logs, Traces, and CloudWatch vs Datadog</title>
      <link>https://jared.lynskey.co.nz/en/posts/2025/2025-10-13-devops-monitoring/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2025/2025-10-13-devops-monitoring/</guid>
      <pubDate>Sun, 12 Oct 2025 00:00:00 +0000</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>DevOps</category>
      <category>Monitoring</category>
      <category>Observability</category>
      <category>CloudWatch</category>
      <category>Datadog</category>
      <category>Developer Experience</category>
      <category>Site Reliability</category>
      <description>Monitoring for developers: how to link metrics, logs and traces, write alerts that say where to start looking, and when CloudWatch is enough and Datadog earns its price.</description>
      <content:encoded><![CDATA[<p>A monitoring system for developers has one job: when something breaks, tell someone fast, and show them what broke and where. That takes metrics, logs and traces linked to each other, alerts that point at a place to start, and dashboards people open without being asked. The tool comes second; on AWS, CloudWatch covers a small team, and Datadog earns its price once you&rsquo;re big.</p>
<p>There&rsquo;s a gap between &ldquo;works on my machine&rdquo; and &ldquo;works in production&rdquo;, and monitoring is supposed to bridge it. Most of the time it doesn&rsquo;t, because it gets built as an ops tool rather than something a developer reaches for when things break.</p>
<p>I&rsquo;ve just gone freelance doing DevOps and infrastructure work, and the monitoring setup is one of the first things I look at anywhere I land — it tells you more about a team than the README does. The failure mode is nearly always the same: the tools exist, but they don&rsquo;t connect to how developers actually work. Someone burns three hours on a bug that production logs would&rsquo;ve solved in five minutes. Failures arrive as support tickets instead of alerts. Or the opposite: 200 alerts a day, most of them noise, so everyone ignores all of them, and the first half hour of every incident is spent working out <em>what</em> broke rather than fixing it.</p>
<p>This is the general version. For the specifics, I&rsquo;ve since written up <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/" >what I actually run today</a> on two small products.</p>

<h2 class="relative group">Metrics, logs, and traces — linked, or useless
    <div id="metrics-logs-and-traces--linked-or-useless" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#metrics-logs-and-traces--linked-or-useless" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Metrics tell you something is wrong (error rate spiked, latency jumped). Logs tell you what happened (stack traces, request payloads, error messages). Traces tell you where (which service, which endpoint, which database call). None of that is news. What matters is whether they&rsquo;re linked: when an alert fires, you should be able to click from the metric to the relevant logs to the trace. If developers are manually correlating timestamps across three tools, the setup has failed, no matter how good each tool looks on paper.</p>
<p>Dashboards deserve the same test. CPU, memory and disk I/O matter for capacity planning, but they don&rsquo;t help anyone debug a 500. The dashboards people open voluntarily show business metrics next to technical ones (signups per hour beside error rate), recent deployments overlaid on the timeline, the top five errors in the last hour with links to logs, and latency percentiles — p50, p95, p99 — rather than averages. If nobody opens a dashboard unprompted, it isn&rsquo;t earning its place.</p>
<p>Alerts, same idea. Every alert should answer two questions: what&rsquo;s broken, and where do I start looking?</p>
<p>Bad alert: &ldquo;High CPU on web-server-3&rdquo;</p>
<p>Good alert: &ldquo;Error rate &gt; 5% on /api/payments since 14:32. Last deploy: 14:15 by @sarah. [View logs] [View trace]&rdquo;</p>
<p>A few things push alerts toward the second kind: alert on deviation from a baseline rather than an arbitrary threshold, route by ownership (payment errors go to the payments team, not the whole org), and group related errors into one Slack message instead of 50. Keep the loop tight, too — the time between &ldquo;something broke&rdquo; and &ldquo;a developer knows&rdquo; should be under a minute, which means real-time log streaming rather than five-minute batches, deploy markers on every dashboard, and tracing that survives service boundaries.</p>

<h2 class="relative group">Build it for the people who&rsquo;ll use it
    <div id="build-it-for-the-people-wholl-use-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#build-it-for-the-people-wholl-use-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The biggest mistake ops teams make is building monitoring for themselves. The fix isn&rsquo;t sophisticated. Sit with a developer during a debugging session and watch where they get stuck. Ask what questions come up mid-incident — &ldquo;which service?&rdquo;, &ldquo;what changed?&rdquo;, &ldquo;what did the request look like?&rdquo;. Find out which metrics actually matter for the product; it&rsquo;s rarely CPU, and usually things like checkout completion rate or upload success rate.</p>
<p>Then remove friction. If instrumenting a new service takes a day, it won&rsquo;t happen. Provide libraries that auto-instrument the frameworks you already use (Django, Express, Spring), copy-paste templates for dashboards and alerts, and self-service metrics so nobody has to file a ticket. And treat the whole thing as code: config in version control, alert rules tested so you know they fire when they should, and multi-region considered early if that&rsquo;s where you&rsquo;re headed.</p>
<p>The same habits keep it alive afterwards. Instrumentation goes through code review like any other production code, and a custom metric gets a test. Every postmortem asks what monitoring would have caught this sooner, and the answer goes into the dashboards and alerts that week rather than into the write-up. And a few times a year I go through what&rsquo;s there and delete what nobody opens, because an alert nobody acts on is training people to ignore the next one.</p>

<h2 class="relative group">Small team on AWS? Start with CloudWatch
    <div id="small-team-on-aws-start-with-cloudwatch" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#small-team-on-aws-start-with-cloudwatch" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you&rsquo;re under 50 engineers and on AWS, CloudWatch is the right starting point, and I&rsquo;d resist buying anything fancier on day one. EC2, Lambda, RDS and ECS all report to it automatically, so you get visibility without writing instrumentation. It runs $10–50 a month for a small team with no platform fee, and metrics, logs, traces (via X-Ray) and alarms all live in one place. You can have meaningful alerts and dashboards up in an afternoon, not weeks.</p>
<p>CloudWatch Logs Insights is the underrated bit — you can query logs without standing up Elasticsearch:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">fields @timestamp, @message
</span></span><span class="line"><span class="cl">| filter @message like /ERROR/
</span></span><span class="line"><span class="cl">| stats count() by bin(5m)</span></span></code></pre></div></div>
<p>Beyond that:</p>
<ul>
<li>Composite alarms cut noise — alert when error rate is high <em>and</em> response time is degraded, not on each condition separately.</li>
<li>Custom metrics are where the real value is. Push business metrics (signups, transactions, feature usage) through the CloudWatch SDK alongside the infrastructure ones.</li>
<li>CloudWatch Synthetics runs scheduled canaries through real user journeys, so you learn the checkout is broken before your users do.</li>
<li>X-Ray gives you distributed tracing with minimal setup — good enough for most microservice architectures.</li>
</ul>
<p>You&rsquo;ll know when you&rsquo;ve outgrown it: you go multi-cloud, you want proper anomaly detection, dashboard customisation starts to hurt, or the team passes 50 engineers and needs real collaboration features.</p>

<h2 class="relative group">Bigger teams: Datadog
    <div id="bigger-teams-datadog" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#bigger-teams-datadog" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Datadog is expensive — $20–100K+ a year for a large org is normal — but at scale it earns its keep. One view across AWS, Azure, GCP, on-prem, containers, serverless, databases and frontend. Watchdog flags anomalies without hand-tuned thresholds, which matters because nobody can manually watch thousands of services. And you get the collaboration layer CloudWatch lacks: team dashboards, RBAC, shared investigation notebooks, PagerDuty/Opsgenie integration, multi-condition alerts, threshold forecasting, maintenance windows, and an APM that goes down to code-level profiling with auto-generated service maps.</p>
<p>Rolling it out, in roughly the order I&rsquo;d do it: instrument critical services first and tag everything by team and environment from day one. Agree naming conventions, dashboard templates, severity levels and SLO definitions early — retrofitting standards is miserable. Wire it into CI/CD deploy markers, incident management and Slack. Budget time for training, because Datadog is powerful but not self-explanatory. And watch the bill: filter noisy metrics, sample traces on high-volume services, and audit which features you&rsquo;re actually paying for.</p>
<p>Worth a look before committing: New Relic (similar, sometimes cheaper for high-volume tracing), Dynatrace (strong AIOps, popular in financial services), Splunk (best-in-class log analysis, especially if security already runs it), and Grafana Cloud (the natural home if you&rsquo;re already on Prometheus/Loki).</p>
<p>In practice a lot of teams land on a hybrid: CloudWatch for AWS-native services because it&rsquo;s automatic and cheap, Datadog for the application layer, with CloudWatch metrics ingested into Datadog for one unified view. It&rsquo;s unglamorous, and it works.</p>

<h2 class="relative group">CloudWatch vs Datadog at a glance
    <div id="cloudwatch-vs-datadog-at-a-glance" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudwatch-vs-datadog-at-a-glance" aria-label="Anchor">#</a>
    </span>
    
</h2>
<table>
	<thead>
			<tr>
					<th></th>
					<th>CloudWatch</th>
					<th>Datadog</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Fits best</td>
					<td>Under ~50 engineers, all on AWS</td>
					<td>Bigger orgs, many services, more than one cloud</td>
			</tr>
			<tr>
					<td>Cost shape</td>
					<td>Pay as you go, no platform fee; roughly $10–50 a month for a small team</td>
					<td>$20–100K+ a year is normal for a large org, and the bill needs watching</td>
			</tr>
			<tr>
					<td>Setup effort</td>
					<td>EC2, Lambda, RDS and ECS report automatically; useful alerts in an afternoon</td>
					<td>Instrument critical services, agree tags and naming early, budget time for training</td>
			</tr>
			<tr>
					<td>Multi-cloud</td>
					<td>Built around AWS; going multi-cloud is a sign you&rsquo;ve outgrown it</td>
					<td>One view across AWS, Azure, GCP, on-prem, containers and serverless</td>
			</tr>
			<tr>
					<td>Anomaly detection</td>
					<td>Mostly alarms you tune yourself; wanting more is another sign</td>
					<td>Watchdog flags anomalies without hand-tuned thresholds</td>
			</tr>
			<tr>
					<td>Collaboration</td>
					<td>Thin; teams past ~50 engineers start to miss it</td>
					<td>Team dashboards, RBAC, shared notebooks, PagerDuty/Opsgenie</td>
			</tr>
			<tr>
					<td>Tracing</td>
					<td>X-Ray, minimal setup</td>
					<td>APM down to code-level profiling, auto-generated service maps</td>
			</tr>
	</tbody>
</table>

<h2 class="relative group">Starting from scratch: the build order
    <div id="starting-from-scratch-the-build-order" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#starting-from-scratch-the-build-order" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The order I&rsquo;d do things in:</p>
<ol>
<li>Talk to developers about what hurts during incidents.</li>
<li>Define SLOs for the flows that matter. What does &ldquo;working&rdquo; mean for login, checkout, search?</li>
<li>Instrument that critical path first.</li>
<li>Add distributed tracing next; it has the best debugging ROI in a microservice setup.</li>
<li>Alert on those flows rather than on CPU, and send each alert to the team that owns it.</li>
<li>Write a runbook for every alert so 3am-you knows what to check.</li>
<li>Put a quarterly review in the calendar to delete stale alerts, fix drifted thresholds, and confirm the dashboards still match the architecture.</li>
</ol>
<p>And the traps I keep pulling teams out of: too many tools (three that work together beat six that don&rsquo;t), graphs with no baseline (is 500 rps normal or a 10x spike?), monitoring things users don&rsquo;t feel (disk I/O on a stateless container), and no redundancy for the monitoring itself — if your alerting dies during an incident, you&rsquo;re blind exactly when you can least afford it.</p>
<p>None of this needs the fanciest tooling. It needs developers to get the information they need, in the place they&rsquo;ll actually look, fast.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2025/2025-10-13-devops-monitoring/featured.jpeg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
