<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Webhooks · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/webhooks/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Tue, 17 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/webhooks/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Cloudflare 520, nginx 499 and Postgres Stalls: Keeping Webhooks Alive</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</guid>
      <pubDate>Tue, 17 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>PostgreSQL</category>
      <category>PgBouncer</category>
      <category>Django</category>
      <category>Cloudflare</category>
      <category>nginx</category>
      <category>Webhooks</category>
      <category>Site Reliability</category>
      <category>Observability</category>
      <category>DevOps</category>
      <description>A 20-second Postgres stall timed out a chatbot&#39;s webhooks. How I held and retried them, and fixed Cloudflare 520 errors, nginx 499s and an exhausted PgBouncer pool.</description>
      <content:encoded><![CDATA[<p>The database behind a chatbot service I look after recently stalled twice, for about 20 seconds each time. Nothing crashed. Postgres came back by itself, the app came back with it, and the error rate barely moved. But the service&rsquo;s job is to take chat messages from a partner&rsquo;s system and acknowledge them quickly, and for those 20 seconds an acknowledgement that normally takes 0.4 s took 10 to 20 s. The partner&rsquo;s system doesn&rsquo;t wait that long. It gave up on those messages, and nothing on our side told anyone.</p>
<p>What caused them is still under investigation. The suspects so far are heavy queries, in particular ones that compare long text fields row by row, and several have been rewritten since. But none is confirmed as the cause yet.</p>
<p>So this post is about the other half of the problem, what the service does while the database isn&rsquo;t answering, and what I built after the stalls: a way to find out when the database stalls, a spool that holds requests on disk until it&rsquo;s back, a replay that runs each one exactly once, and a pass over every timeout between the internet and Postgres. A few days later it got a real test I hadn&rsquo;t planned.</p>
<p>If you landed here from one of these errors, each has its own section: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" >Cloudflare 520</a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" ><code>server closed the connection unexpectedly</code></a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#nginx-499-client-closed-request-replies-nobody-received" >nginx 499</a> and <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#pgbouncer-pool-exhausted-the-real-test" >a PgBouncer pool that ran dry</a>.</p>
<figure><div class="film-pair">
  <div class="mc-film film-dark">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRooAAABXRUJQVlA4IH4AAAAQBQCdASogABIAPw1urEqmIyIYDAQAYxDE9IAECBxAGNSFBxYLELF/yzl596pm6x2gAP7r1B4uhu6S+JqEfZ/87AtHGoj32Tp5yLTqb/5AlWmTYy/0oE8Nrih94UDfWuilEh2jC+eSMIqPA7MB5pMgkX94ddaOOys6I/gAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <div class="mc-film film-light">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light-poster.jpg" style="background:url('data:image/webp;base64,UklGRowAAABXRUJQVlA4IIAAAAAQBQCdASogABIAPw12qkmmI6GYDAQAYxDE9IAHOC3BZAFaEzG+kH6nxw8hG6nIiLJgAP7zbqngFyuj4/0DXJ2xZhiK6QUnRz4fsBZ+3lqB7T1fFerRg31UlZF+A/PCzrCTelVcbpcgNZFBnWzNRBEskzdVca60ciOypAYYRhAAAA==') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script></div><figcaption>This post as an eight-minute film, if you&rsquo;d rather watch than read. Some footage from Coverr.</figcaption>
</figure>


<h2 class="relative group">The requests
    <div id="the-requests" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-requests" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The partner&rsquo;s system sends each new chat message to a webhook endpoint. We answer <code>202 Accepted</code> with a job ID, a background worker generates the bot&rsquo;s reply, and the reply is posted back to a URL the partner gave us. Creating the job is one row in Postgres, so the 202 needs the database. Normally that takes about 0.4 s, and 2.6 s at the 99th percentile.</p>
<p>There&rsquo;s a second kind of client too: a browser extension that asks for a reply and waits for it in the same request, for up to 60 seconds.</p>
<p>The stack is ordinary. Cloudflare in front, nginx, Django, PgBouncer in transaction mode, and Postgres replicated across two machines in different places, with HAProxy sending connections to whichever one is the primary. The app server that takes this traffic runs next to the replica, so its writes cross the internet to the primary. That detail matters later.</p>

<h2 class="relative group">Detecting a Postgres stall before your users do
    <div id="detecting-a-postgres-stall-before-your-users-do" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#detecting-a-postgres-stall-before-your-users-do" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The first fix was the cheapest: know when it happens. The checks we had watched uptime, connection counts and locks, and a 20-second stall tripped none of them. Requests just took longer.</p>
<p>So I added a probe that runs on the host, outside the app, and does what the app does: it connects through the same proxy and commits a transaction, every two seconds. <code>txid_current()</code> makes Postgres assign a transaction ID, which makes it a write without touching any table. Normally it takes about 5 ms. Two commits slower than 2 s is a warning; one slower than 10 s, failing, or hanging is critical.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">psycopg</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">DSN</span> <span class="o">=</span> <span class="s2">&#34;postgresql://monitor@db-proxy:5432/app&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">commit_time</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">psycopg</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">DSN</span><span class="p">,</span> <span class="n">connect_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SET statement_timeout = &#39;15s&#39;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SELECT txid_current()&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">took</span> <span class="o">=</span> <span class="n">commit_time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database commit failed: </span><span class="si">{</span><span class="n">exc</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">took</span> <span class="o">&gt;=</span> <span class="mi">10</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database stalled: a commit took </span><span class="si">{</span><span class="n">took</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> s&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The same script reads the web server&rsquo;s access log and checks how long the 202s took in the last minute: three over 5 s is a warning, any over 10 s is critical. That&rsquo;s the number the partner actually feels.</p>
<p>Two choices in it are deliberate. It runs on the host, not inside the app, so it still works when the app is the thing that&rsquo;s stuck. And it sends alerts straight to the chat channel, not through the app&rsquo;s own alerting, for the same reason. Each condition has a 15-minute cooldown and sends a message when it clears, so a flapping database doesn&rsquo;t flood the channel.</p>

<h2 class="relative group">Holding webhook requests on disk while the database is stuck
    <div id="holding-webhook-requests-on-disk-while-the-database-is-stuck" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#holding-webhook-requests-on-disk-while-the-database-is-stuck" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Knowing about a stall doesn&rsquo;t save the requests in it. For that, the API has to answer the partner within its patience even when the database can&rsquo;t.</p>
<p>Django middleware does it. It runs the real view in a thread and waits up to a budget of 4 seconds, which is well above the normal 99th percentile. If the view finishes in time, its response goes back as usual. If it doesn&rsquo;t, the middleware writes the raw request to a spool directory on disk and answers with a 202 of its own:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span> <span class="nt">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="nt">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span> <span class="nt">&#34;held&#34;</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span></span></span></code></pre></div></div>
<p>The view&rsquo;s thread keeps running, and if the database comes back quickly it finishes the job itself. The partner&rsquo;s documentation says to treat a held 202 exactly like a normal one and not to resend it.</p>
<p>Here&rsquo;s the shape of it, cut down:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SPOOL</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s2">&#34;/var/spool/myapp/held&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">BUDGET_SECONDS</span> <span class="o">=</span> <span class="mf">4.0</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">HoldOnStallMiddleware</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">get_response</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span> <span class="o">=</span> <span class="n">get_response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">path</span> <span class="o">!=</span> <span class="s2">&#34;/api/jobs/&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">body</span>  <span class="c1"># read it before another thread can</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span><span class="o">.</span><span class="n">hex</span>
</span></span><span class="line"><span class="cl">        <span class="n">arrived</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">result</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">def</span> <span class="nf">run</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">worker</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">run</span><span class="p">,</span> <span class="n">daemon</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BUDGET_SECONDS</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="s2">&#34;response&#34;</span> <span class="ow">in</span> <span class="n">result</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">SPOOL</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">parents</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">held</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;key&#34;</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="p">,</span> <span class="s2">&#34;arrived&#34;</span><span class="p">:</span> <span class="n">arrived</span><span class="p">,</span> <span class="s2">&#34;body&#34;</span><span class="p">:</span> <span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">()}</span>
</span></span><span class="line"><span class="cl">        <span class="p">(</span><span class="n">SPOOL</span> <span class="o">/</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">arrived</span><span class="si">:</span><span class="s2">.6f</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="si">}</span><span class="s2">.json&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">write_text</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">held</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="s2">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="s2">&#34;held&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">202</span><span class="p">)</span></span></span></code></pre></div></div>
<p>It sits ahead of every middleware that touches the database, including sessions and authentication, or those would stall first and the budget would never start.</p>

<h2 class="relative group">Retrying held webhooks exactly once
    <div id="retrying-held-webhooks-exactly-once" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#retrying-held-webhooks-exactly-once" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A separate process drains the spool. It waits until <code>SELECT 1</code> answers, then replays the held requests oldest first through the real endpoint, with a signed header that carries the hold key and the original arrival time. A 202 removes the file. A 429 or a 5xx means try again later, up to six hours. Anything else moves the file to a <code>failed/</code> folder for a person to look at. A lock file keeps it to one drainer per host, and it exits every hour so the next one always runs the current code.</p>
<p>The hard part is &ldquo;exactly once&rdquo;. When the database comes back, two things may try to create the same job: the original request thread, which was still waiting, and the drainer. The obvious answer is to hand the request over in memory, so the thread stops and leaves it to the drainer. A stall test in a sandbox showed why that doesn&rsquo;t work: the stall can start after the thread has already sent its <code>INSERT</code>. The thread can&rsquo;t take back a write it&rsquo;s waiting on.</p>
<p>So the database decides. Every job stores its hold key under a partial unique index, and whichever insert comes second fails:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">IntegrityError</span><span class="p">,</span> <span class="n">models</span><span class="p">,</span> <span class="n">transaction</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Job</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">hold_key</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">blank</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">payload</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">JSONField</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">constraints</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">            <span class="n">models</span><span class="o">.</span><span class="n">UniqueConstraint</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;hold_key&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">                <span class="n">condition</span><span class="o">=</span><span class="n">models</span><span class="o">.</span><span class="n">Q</span><span class="p">(</span><span class="n">hold_key__isnull</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">                <span class="n">name</span><span class="o">=</span><span class="s2">&#34;job_hold_key_uniq&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_job</span><span class="p">(</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">with</span> <span class="n">transaction</span><span class="o">.</span><span class="n">atomic</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">IntegrityError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">)</span></span></span></code></pre></div></div>
<p>If creating a job also cancels an older one (ours does, when a newer request arrives for the same webhook), both statements go in one transaction, so the loser of the race cancels nothing.</p>
<p>Arrival time matters too. A held request is judged by when it arrived, not when it was replayed. It only replaces jobs created before it arrived, and if a newer request for the same webhook was queued while it sat on disk, it&rsquo;s dropped as superseded rather than run late.</p>

<h2 class="relative group">The first false alarm
    <div id="the-first-false-alarm" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-first-false-alarm" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Within hours of going live, the spool was holding three requests an hour, with no stall anywhere. Each was replayed within seconds, and each sent a warning and a summary to the ops channel, which buried anything real.</p>
<p>The cause was in front of the 202. For some requests the view described an image before creating the job, which took 1.6 to 2.1 s for an image it hadn&rsquo;t seen before, and occasionally pushed the acknowledgement past 4 s. The worker describes the image again anyway, so the early call only made the 202 slower. It moved into the worker, and the drainer now alerts only when a held request has waited 30 seconds without being replayed.</p>
<p>A spool with a short budget turns out to be a good way to find everything slow that sits in front of an acknowledgement.</p>

<h2 class="relative group">Finding the failing hop with Cloudflare analytics
    <div id="finding-the-failing-hop-with-cloudflare-analytics" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#finding-the-failing-hop-with-cloudflare-analytics" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Our own logs only see requests that reach the app. When something fails between the client and Django, the app has nothing to say about it. Cloudflare sees every request first, and it can tell you which side failed.</p>
<p>The zone is on the free plan, which has no Logpush, but the GraphQL Analytics API works on it. It gives request counts by edge status over time, percentiles of <code>originResponseDurationMs</code> (how long our side took to answer Cloudflare, including the proxy and the network), and sampled records of individual requests with both the status Cloudflare returned and the status our origin returned:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -s https://api.cloudflare.com/client/v4/graphql <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Authorization: Bearer </span><span class="nv">$CF_ANALYTICS_TOKEN</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  --data @- <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">{&#34;query&#34;: &#34;query($zone: String!, $since: Time!) { viewer { zones(filter: {zoneTag: $zone}) { httpRequestsAdaptive(limit: 50, orderBy: [datetime_DESC], filter: {datetime_geq: $since, edgeResponseStatus_geq: 499}) { datetime edgeResponseStatus originResponseStatus originResponseDurationMs clientRequestPath } } } }&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s"> &#34;variables&#34;: {&#34;zone&#34;: &#34;YOUR_ZONE_ID&#34;, &#34;since&#34;: &#34;YOUR_START_TIME&#34;}}
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span></span></span></code></pre></div></div>
<p><code>YOUR_START_TIME</code> is an ISO 8601 time in UTC. The Cloudflare analytics API token only needs the zone&rsquo;s Analytics Read permission. I put the results on an internal page next to the app&rsquo;s own timings.</p>
<p>Three things made it useful.</p>
<p><strong>Origin time against our own time.</strong> Cloudflare&rsquo;s origin time includes everything after the request leaves Cloudflare; the app&rsquo;s request traces cover only our code. The first reading was an origin median of 730 to 895 ms against about 500 ms of handling in the app, so a few hundred milliseconds were going somewhere before our code ran.</p>
<p><strong>The errors that never reach your logs.</strong> A 499 means the client hung up before we answered. 520 to 526 mean Cloudflare couldn&rsquo;t get a good answer from the origin. Neither shows up in Django. (A 504 from Cloudflare can be a different story again: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >some aren&rsquo;t real</a>.)</p>
<p><strong>The <code>CF-Ray</code> header.</strong> Cloudflare adds it to every request it forwards. The app now stores it on each request&rsquo;s trace, so a request Cloudflare saw can be matched to one the app handled. An edge error with no matching trace never reached the app, which narrows the problem to the hops in between.</p>
<p>That last test found two problems.</p>

<h3 class="relative group">Cloudflare 520 error fix: nginx keepalive_timeout longer than Cloudflare&rsquo;s
    <div id="cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Cloudflare returned three 520s on the webhook endpoint one day, the error its page describes as &ldquo;Web server is returning an unknown error&rdquo;, and none of them had a trace in the app. So it was the proxy.</p>
<p>Cloudflare keeps idle connections to your origin open and reuses them for up to 900 seconds. nginx was closing idle keepalive connections after 90. Now and then Cloudflare sent a request down a connection at the moment nginx closed it, and got nothing back. The fix is one line:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="c1"># Longer than Cloudflare reuses an idle connection to the origin (900 s)
</span></span></span><span class="line"><span class="cl"><span class="k">keepalive_timeout</span> <span class="s">930s</span><span class="p">;</span></span></span></code></pre></div></div>

<h3 class="relative group">server closed the connection unexpectedly (Postgres): HAProxy and PgBouncer timeouts
    <div id="server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>It was the same kind of bug I&rsquo;d fixed lower down months before. HAProxy&rsquo;s 30-second timeouts were closing PgBouncer&rsquo;s idle connections to Postgres, and the app saw <code>server closed the connection unexpectedly</code>. HAProxy now allows 600 seconds, longer than PgBouncer keeps an idle server connection (300), and Django checks a reused connection before handing it out (<code>CONN_HEALTH_CHECKS = True</code>). The rule is the same at both layers: whoever holds an idle connection open must keep it longer than whoever reuses it.</p>

<h3 class="relative group">nginx 499 (client closed request): replies nobody received
    <div id="nginx-499-client-closed-request-replies-nobody-received" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#nginx-499-client-closed-request-replies-nobody-received" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When a client hangs up, Django still finishes the request and records a 200. Only nginx knows the client was gone, and it logs a 499. Logging those separately, with the request ID the app also uses, showed one office losing about 60% of its replies in the browser extension, to page reloads in the middle of a reply.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="k">map</span> <span class="nv">$status</span> <span class="nv">$client_gone</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kn">499</span>     <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">default</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">log_format</span> <span class="s">client_aborts</span> <span class="s">escape=json</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;</span><span class="p">{</span><span class="kn">&#34;time&#34;:&#34;$time_iso8601&#34;,&#34;request_id&#34;:&#34;$request_id&#34;,&#39;</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;&#34;uri&#34;:&#34;</span><span class="nv">$uri&#34;,&#34;waited&#34;:$request_time}&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">access_log</span> <span class="s">/var/log/nginx/client-aborts.log</span> <span class="s">client_aborts</span> <span class="s">if=</span><span class="nv">$client_gone</span><span class="p">;</span></span></span></code></pre></div></div>
<p>A small job reads that file every 15 seconds and attaches each abort to its request trace once the request has finished, so a lost reply has a person and a conversation against it. An abort after 55 seconds or more is the extension&rsquo;s own 60-second timeout; anything shorter was usually a page reload.</p>
<p>One thing that doesn&rsquo;t fix a 499 is raising nginx&rsquo;s timeouts. The client is the one that gave up, so the fix is on the client&rsquo;s side or in answering sooner, and the useful part on the server is knowing which replies were lost.</p>

<h2 class="relative group">PgBouncer pool exhausted: the real test
    <div id="pgbouncer-pool-exhausted-the-real-test" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pgbouncer-pool-exhausted-the-real-test" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Then someone opened a statistics page. It sent one request per person on the team, 69 at once, and each pulled every row of that person&rsquo;s history through PgBouncer to the primary, across the internet. The pool has 50 connections plus 20 in reserve, and every one of them was taken.</p>
<p>For 1 minute 46 seconds, nothing that needed the database could get it. The spool held 80 webhook requests. All 80 were accounted for afterwards: 74 delivered, and 6 dropped because a newer request for the same webhook had replaced them. The browser extension has no spool, and 14 of its replies were lost.</p>
<p>So the spool worked, and the cause was ours. The fixes were about keeping heavy reads away from the path that serves requests:</p>
<ul>
<li><strong>One request instead of 69.</strong> The page asks for everyone at once, and the counts and averages are worked out in SQL instead of in Python over every row.</li>
<li><strong>Store what you&rsquo;ll want to average.</strong> The page scored each row by comparing two long text fields, in Python, for every row, which is why it pulled every row&rsquo;s full text across the internet. The score is now saved when the row is saved. For older rows, pairs that are identical count as a perfect score in SQL, and only the 2% or so that differ are fetched and compared. A timestamp a response-time query used to dig out of each conversation&rsquo;s history is saved when the row is created. That query used to scan up to 200 earlier messages per row; on a two-day window it went from 19.3 s to 1.5 s.</li>
<li><strong>Read from the node on the same machine.</strong> Reports get their own database connection to whichever Postgres node runs on the same host, read-only, with a statement timeout. If it&rsquo;s a replica more than five minutes behind, they use the normal connection instead.</li>
<li><strong>A cap.</strong> At most two report queries run at once per process; the rest wait up to 60 seconds and then get a 503.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># settings.py: a separate connection for heavy reads</span>
</span></span><span class="line"><span class="cl"><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;reports&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="o">**</span><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;HOST&#34;</span><span class="p">:</span> <span class="s2">&#34;localhost&#34;</span><span class="p">,</span>  <span class="c1"># the Postgres node on this machine</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;CONN_MAX_AGE&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;OPTIONS&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;connect_timeout&#34;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;options&#34;</span><span class="p">:</span> <span class="s2">&#34;-c statement_timeout=120s -c default_transaction_read_only=on&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># reports.py: at most two heavy reads per process</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">_slots</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">BoundedSemaphore</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">heavy_read</span><span class="p">(</span><span class="n">view</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="nd">@wraps</span><span class="p">(</span><span class="n">view</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">_slots</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="s2">&#34;Reports are busy. Try again shortly.&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">_slots</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">wrapper</span></span></span></code></pre></div></div>
<p>One mistake in the first version is worth repeating. A failed report read was retried once on the normal connection, which made sense for a lost connection. But it also did it for a query that had hit its statement timeout, so the slowest query on the page ran for another 120 seconds on the primary, through the shared pool. Now only a lost connection falls back, a replica conflict is retried on the same node, and a timeout is reported as it is.</p>

<h2 class="relative group">Every timeout from Cloudflare to Postgres
    <div id="every-timeout-from-cloudflare-to-postgres" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-timeout-from-cloudflare-to-postgres" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most of what went wrong was two settings that only make sense together, set in different places by different people at different times. So here they all are, from the outside in:</p>
<table>
	<thead>
			<tr>
					<th>Hop</th>
					<th>Setting</th>
					<th>Value</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Partner&rsquo;s system</td>
					<td>Waits for our 202</td>
					<td>a few seconds (the exact limit is still under investigation)</td>
			</tr>
			<tr>
					<td>Our 202</td>
					<td>Hold budget before spooling</td>
					<td>4 s (normal p99 2.6 s)</td>
			</tr>
			<tr>
					<td>Browser extension</td>
					<td>Gives up on a reply</td>
					<td>60 s</td>
			</tr>
			<tr>
					<td>Cloudflare → nginx</td>
					<td>Reuses an idle connection</td>
					<td>up to 900 s</td>
			</tr>
			<tr>
					<td>nginx</td>
					<td><code>keepalive_timeout</code></td>
					<td>930 s</td>
			</tr>
			<tr>
					<td>Cloudflare</td>
					<td>Waits for the origin&rsquo;s response</td>
					<td>100 s (then a 524, &ldquo;A timeout occurred&rdquo;)</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td><code>timeout client</code> / <code>timeout server</code></td>
					<td>600 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>server_idle_timeout</code></td>
					<td>300 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>query_wait_timeout</code> (waiting for a free connection)</td>
					<td>120 s</td>
			</tr>
			<tr>
					<td>Django</td>
					<td><code>CONN_MAX_AGE</code>, <code>CONN_HEALTH_CHECKS</code></td>
					<td>60 s, on</td>
			</tr>
			<tr>
					<td>Report connection</td>
					<td><code>statement_timeout</code>, <code>connect_timeout</code></td>
					<td>120 s, 3 s</td>
			</tr>
	</tbody>
</table>
<p>One pair in that table is still under investigation. If the report pages are served through Cloudflare&rsquo;s proxy, their 120-second statement timeout outlasts Cloudflare&rsquo;s 100 seconds, and a slow report would end in a 524 from Cloudflare before Postgres gives up on it.</p>
<p>Two rules cover nearly all of it. Idle timeouts should get longer as you go inwards: each hop keeps an idle connection open longer than the hop in front of it might reuse it. Waiting times should get shorter as you go inwards: each hop gives up, or answers something useful, before the one in front of it gives up on it. The 4-second hold budget is the second rule applied to the 202. Not every row follows it yet: PgBouncer will still queue a request for a free connection for 120 seconds, longer than either client waits.</p>

<h2 class="relative group">What I&rsquo;d do first next time
    <div id="what-id-do-first-next-time" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-do-first-next-time" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If I were setting this up again, I&rsquo;d start with the probe, because it costs an hour and tells you what you&rsquo;re dealing with. Then the Cloudflare analytics with <code>CF-Ray</code> on every trace, because it tells you which hop to look at. The spool came third, and it&rsquo;s the part I&rsquo;m happiest with, but it only made sense once I knew stalls were real and how long they lasted.</p>
<p>I&rsquo;d also take the browser extension&rsquo;s 14 lost replies as a hint. It has no spool because it waits for its answer in the same request, so there&rsquo;s nothing to hold. Giving it a job ID and a way to pick the answer up later would let it ride out the next stall the same way the webhooks did.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Upload Directly to S3 and Still Compress Images: Tigris Object Notifications with Django</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/</guid>
      <pubDate>Sat, 07 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Tigris</category>
      <category>S3</category>
      <category>S3 Event Notifications</category>
      <category>Presigned URLs</category>
      <category>Image Uploads</category>
      <category>Image Compression</category>
      <category>Object Storage</category>
      <category>Webhooks</category>
      <category>Django</category>
      <category>Fly.io</category>
      <category>Discord</category>
      <description>Direct-to-S3 uploads with presigned URLs are fast, but your backend never hears that the file arrived, so compression and thumbnails get awkward. Tigris object notifications fix that: the bucket POSTs to a Django webhook and the photo is processed while the user is still typing. The same signal turns log files into Discord alerts.</description>
      <content:encoded><![CDATA[<p>Every image upload I&rsquo;ve built has had the same tension in it. The person uploading wants the photo to go up fast. I want the server to compress it, make a thumbnail, pull a blurhash and a couple of colours out of it, drop the metadata that shouldn&rsquo;t be there, and store it the way the rest of the app expects. The easy way to get my list is to send the file through the API, and that is exactly what makes their list slow.</p>
<p>The fast way is a presigned URL. The API signs a URL, the client PUTs the file straight to the bucket, and the bytes never touch the API at all. It&rsquo;s the standard answer for good reason. But the moment you do it you&rsquo;ve cut your backend out of the loop. It handed out a URL and then heard nothing. Did the file arrive? Is it the size you expected? Is it even a JPEG? Something has to tell the backend, and the usual options are all a bit sad: trust the client to call back and say &ldquo;done&rdquo; (clients lie, crash and lose signal), poll the bucket on a schedule (slow and wasteful), or process lazily on the first read (the first viewer pays, and you still don&rsquo;t know when the file is safe to reference). This is what makes uploading directly to S3 look harder than it is. The upload is easy. The signalling is what&rsquo;s missing.</p>
<p>Object notifications are that signal. Tigris, the S3-compatible storage I use on Fly.io, will POST a small JSON body to your app every time an object is created or deleted in a bucket. On AWS the equivalent is S3 event notifications, which usually means wiring up a Lambda, an SQS queue or EventBridge before anything reaches your code (SNS can POST to an HTTPS endpoint, but then you&rsquo;re confirming subscriptions and verifying signatures first). Tigris keeps it to one HTTP request to a URL you own, which is about as simple as storage-to-backend signalling gets. The feature exists so the bucket can talk to your backend without you building a bridge for it.</p>
<p>This post is the whole flow: the setup, a Django endpoint that receives the events, a presigned upload that keeps every bit of server-side processing, and the gotchas (CORS, size limits, expiry, multiple files). At the end, the same signal doing back-office work, where log files landing in the bucket became Discord alerts and let me delete a cron job I&rsquo;d been nursing for months.</p>

<h2 class="relative group">The short answer
    <div id="the-short-answer" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-short-answer" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In the Tigris dashboard, open the bucket, then Settings, then Object notifications. Give it an HTTPS URL on your app and pick basic auth (a username and password Tigris sends in the <code>Authorization</code> header; a bearer token is the other option). Add a filter so you only hear about the keys you care about. The filter is a SQL-like expression, so &ldquo;only the three prefixes this post handles&rdquo; is:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">WHERE</span><span class="w"> </span><span class="o">`</span><span class="k">key</span><span class="o">`</span><span class="w"> </span><span class="n">REGEXP</span><span class="w"> </span><span class="s2">&#34;^(uploads/raw|logs/api|media)/&#34;</span></span></span></code></pre></div></div>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="Tigris bucket settings, Notifications tab: object notifications switched on, a webhook URL, basic authentication with a username and password, and a SQL-like key filter"
    width="1204"
    height="671"
    src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_7e9da7c6ca1125db.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_7e9da7c6ca1125db.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_4ddc968851d1f052.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_4ddc968851d1f052.webp"></span><figcaption>The whole setup: an HTTPS URL on your app, basic auth, and a filter on the key. The bucket name and URL here are swapped for examples.</figcaption></figure>
</p>
<p>Tigris then sends a batch like this on every change:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;events&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;eventName&#34;</span><span class="p">:</span> <span class="s2">&#34;OBJECT_CREATED_PUT&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;eventTime&#34;</span><span class="p">:</span> <span class="s2">&#34;2026-11-07T09:12:44Z&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;bucket&#34;</span><span class="p">:</span> <span class="s2">&#34;example-bucket&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;object&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;key&#34;</span><span class="p">:</span> <span class="s2">&#34;uploads/raw/1f3c.jpg&#34;</span><span class="p">,</span> <span class="nt">&#34;size&#34;</span><span class="p">:</span> <span class="mi">48213</span><span class="p">,</span> <span class="nt">&#34;eTag&#34;</span><span class="p">:</span> <span class="s2">&#34;9b2c…&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The event names you&rsquo;ll meet are <code>OBJECT_CREATED_PUT</code>, <code>OBJECT_CREATED_MULTIPART</code>, <code>OBJECT_CREATED_COPY</code> and <code>OBJECT_DELETED</code>, plus rename and storage-tier transitions. Your endpoint checks the auth, routes each event by key prefix, and returns a 2xx within ten seconds. Three things to plan for from the start: delivery is at least once, so you will see repeats; events are batched, so one request can carry several; and they can arrive out of order, so don&rsquo;t read &ldquo;latest&rdquo; off the order they came in.</p>

<h2 class="relative group">A Django webhook for the bucket&rsquo;s events
    <div id="a-django-webhook-for-the-buckets-events" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-django-webhook-for-the-buckets-events" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The view is boring on purpose. It checks the auth, parses the body and hands each event to a router:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">base64</span><span class="o">,</span> <span class="nn">hmac</span><span class="o">,</span> <span class="nn">json</span><span class="o">,</span> <span class="nn">logging</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.views.decorators.csrf</span> <span class="kn">import</span> <span class="n">csrf_exempt</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.views.decorators.http</span> <span class="kn">import</span> <span class="n">require_POST</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">.bucket_events</span> <span class="kn">import</span> <span class="n">handle</span><span class="p">,</span> <span class="n">parse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">_authorised</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">secret</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">header</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;Authorization&#34;</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">header</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;Basic &#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">password</span> <span class="o">=</span> <span class="n">base64</span><span class="o">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">header</span><span class="p">[</span><span class="mi">6</span><span class="p">:])</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="p">(</span><span class="ne">ValueError</span><span class="p">,</span> <span class="ne">UnicodeDecodeError</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">hmac</span><span class="o">.</span><span class="n">compare_digest</span><span class="p">(</span><span class="n">user</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span> <span class="sa">b</span><span class="s2">&#34;tigris&#34;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">hmac</span><span class="o">.</span><span class="n">compare_digest</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">password</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span> <span class="n">secret</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@csrf_exempt</span>
</span></span><span class="line"><span class="cl"><span class="nd">@require_POST</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">bucket_webhook</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">secret</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">BUCKET_WEBHOOK_SECRET</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">secret</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;not configured&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">_authorised</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">secret</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;unauthorised&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">401</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">body</span> <span class="ow">or</span> <span class="sa">b</span><span class="s2">&#34;</span><span class="si">{}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;invalid json&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">400</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">event</span> <span class="ow">in</span> <span class="n">parse</span><span class="p">(</span><span class="n">body</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">handle</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">logger</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">&#34;bucket event failed: </span><span class="si">%s</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;ok&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span></span></span></code></pre></div></div>
<p><code>compare_digest</code> is there so the check takes the same time whatever you send it (on bytes, because on strings it raises for non-ASCII input, and a crafted header would get a 500 instead of a 401). If the secret isn&rsquo;t set, say on a fresh staging box, the endpoint refuses everything rather than accepting everything. And a handler that throws is logged and still answered 200: Tigris retries anything that isn&rsquo;t a 2xx, a retry would hit the same bug, and by then the event has been claimed (two sections down) so the retry would do nothing anyway. The log line is what gets it fixed. That assumes the failure is deterministic. A transient storage timeout consumes the event for good: the photos don&rsquo;t mind, because the save can process them itself, but a log file&rsquo;s alert would be lost. If that matters to you, release the claim in the <code>except</code> and answer 500 so Tigris retries, and accept that a real bug then retries until it&rsquo;s fixed.</p>

<h2 class="relative group">Routing by prefix
    <div id="routing-by-prefix" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#routing-by-prefix" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Everything in the bucket is already organised by prefix, so that&rsquo;s the natural way to route. Each use is one function and one line in a table:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">HANDLERS</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;uploads/raw/&#34;</span><span class="p">,</span>   <span class="n">on_raw_photo</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;logs/api/&#34;</span><span class="p">,</span>      <span class="n">on_api_log_file</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;media/&#34;</span><span class="p">,</span>         <span class="n">on_media_change</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">fn</span> <span class="ow">in</span> <span class="n">HANDLERS</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">prefix</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">claim</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>  <span class="c1"># see below</span>
</span></span><span class="line"><span class="cl">                <span class="n">fn</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span></span></span></code></pre></div></div>
<p>Adding a fourth thing later is a new function and a new row. I like that more than I expected to.</p>

<h2 class="relative group">At least once means &ldquo;maybe twice&rdquo;
    <div id="at-least-once-means-maybe-twice" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#at-least-once-means-maybe-twice" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Tigris retries if your endpoint is slow or returns an error, and like most webhook senders it makes no promise that you&rsquo;ll see an event only once. So before a handler runs, the event is claimed by <code>(key, eTag, eventName)</code>. The eTag matters: if a file is overwritten, it&rsquo;s a new version with a new eTag and should be handled again. A repeat of the same version shouldn&rsquo;t.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.core.cache</span> <span class="kn">import</span> <span class="n">cache</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">claim</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">ttl</span><span class="o">=</span><span class="mi">7</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">k</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;bucket-event:</span><span class="si">{</span><span class="n">event</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">event</span><span class="o">.</span><span class="n">etag</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">ttl</span><span class="p">)</span>  <span class="c1"># False if someone already claimed it</span></span></span></code></pre></div></div>
<p><code>cache.add</code> only sets the key if it isn&rsquo;t there already, which in Redis is atomic, so two machines receiving the same retry can&rsquo;t both win. I use a database-backed lease for this in the real thing because I wanted it to survive a Redis flush, but the idea is identical.</p>

<h2 class="relative group">Direct-to-S3 image uploads, with the pipeline intact
    <div id="direct-to-s3-image-uploads-with-the-pipeline-intact" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#direct-to-s3-image-uploads-with-the-pipeline-intact" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is the part that pays for the plumbing.</p>
<p>Previously, picking photos for a listing meant the app held them, and when you hit Post it sent them all to the API as base64. The API resized them, made thumbnails, worked out colours and a blurhash, and only then replied. Ten photos on a slow connection was a long spinner. And a request carrying ten base64 photos is a big, slow body for an API server to hold in memory, which was quietly why I kept photos small.</p>
<p>Now it&rsquo;s four steps, and the backend is told about every one of them.</p>

<h3 class="relative group">1. The client shrinks the photo and asks for a slot
    <div id="1-the-client-shrinks-the-photo-and-asks-for-a-slot" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#1-the-client-shrinks-the-photo-and-asks-for-a-slot" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The app still resizes on the device, to the longest edge the server keeps (1080px) as a JPEG at 0.7 quality. That resize is for the user&rsquo;s bandwidth, nothing else. Then it asks the API for an upload slot:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.utils</span> <span class="kn">import</span> <span class="n">timezone</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s2">&#34;s3&#34;</span><span class="p">,</span> <span class="n">endpoint_url</span><span class="o">=</span><span class="s2">&#34;https://fly.storage.tigris.dev&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">BUCKET</span> <span class="o">=</span> <span class="s2">&#34;example-bucket&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_upload_slot</span><span class="p">(</span><span class="n">user</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">public_id</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">key</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;uploads/raw/</span><span class="si">{</span><span class="n">public_id</span><span class="si">}</span><span class="s2">.jpg&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span> <span class="o">=</span> <span class="n">PhotoUpload</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">public_id</span><span class="o">=</span><span class="n">public_id</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">raw_key</span><span class="o">=</span><span class="n">key</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">expires_at</span><span class="o">=</span><span class="n">timezone</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">2</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;put_object&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">Params</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Bucket&#34;</span><span class="p">:</span> <span class="n">BUCKET</span><span class="p">,</span> <span class="s2">&#34;Key&#34;</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="s2">&#34;ContentType&#34;</span><span class="p">:</span> <span class="s2">&#34;image/jpeg&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">15</span> <span class="o">*</span> <span class="mi">60</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;upload_id&#34;</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;url&#34;</span><span class="p">:</span> <span class="n">url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;method&#34;</span><span class="p">:</span> <span class="s2">&#34;PUT&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;headers&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;image/jpeg&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;expires_in&#34;</span><span class="p">:</span> <span class="mi">15</span> <span class="o">*</span> <span class="mi">60</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span></span></span></code></pre></div></div>
<p>The slot is a row before it&rsquo;s a file. That&rsquo;s what lets the server know who asked for it, when it expires, and later whether it was ever used.</p>

<h3 class="relative group">2. The client PUTs straight to the bucket
    <div id="2-the-client-puts-straight-to-the-bucket" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#2-the-client-puts-straight-to-the-bucket" aria-label="Anchor">#</a>
    </span>
    
</h3>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">slot</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">api</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="s2">&#34;/photo-uploads/&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">res</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">slot</span><span class="p">.</span><span class="nx">url</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">method</span>: <span class="kt">slot.method</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">headers</span>: <span class="kt">slot.headers</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">body</span>: <span class="kt">blob</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">res</span><span class="p">.</span><span class="nx">ok</span><span class="p">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">&#34;upload refused&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="c1">// keep slot.upload_id for the save
</span></span></span></code></pre></div></div>
<p>The API never sees the bytes. This is the same from a browser, from React Native or from a native app; the only difference is that a browser needs a CORS rule on the bucket, which is in the gotchas below.</p>

<h3 class="relative group">3. Tigris tells the backend, and the pipeline runs
    <div id="3-tigris-tells-the-backend-and-the-pipeline-runs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#3-tigris-tells-the-backend-and-the-pipeline-runs" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Within a few seconds Tigris posts an <code>OBJECT_CREATED_PUT</code> for <code>uploads/raw/…</code>, and the handler runs the same image pipeline the base64 path always used, while the person is still typing their title:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">BytesIO</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span><span class="p">,</span> <span class="n">ImageOps</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.core.files.base</span> <span class="kn">import</span> <span class="n">ContentFile</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.utils</span> <span class="kn">import</span> <span class="n">timezone</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">blurhash</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">MAX_BYTES</span> <span class="o">=</span> <span class="mi">15</span> <span class="o">*</span> <span class="mi">1024</span> <span class="o">*</span> <span class="mi">1024</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">on_raw_photo</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">event</span><span class="o">.</span><span class="n">name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;OBJECT_CREATED&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">event</span><span class="o">.</span><span class="n">size</span> <span class="o">&gt;</span> <span class="n">MAX_BYTES</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">s3</span><span class="o">.</span><span class="n">delete_object</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="n">Key</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span> <span class="o">=</span> <span class="n">PhotoUpload</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">raw_key</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">processed_at</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">upload</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="n">data</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">get_object</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="n">Key</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">)[</span><span class="s2">&#34;Body&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">process</span><span class="p">(</span><span class="n">upload</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">to_webp</span><span class="p">(</span><span class="n">im</span><span class="p">,</span> <span class="n">longest</span><span class="p">,</span> <span class="n">quality</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">im</span> <span class="o">=</span> <span class="n">im</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">im</span><span class="o">.</span><span class="n">thumbnail</span><span class="p">((</span><span class="n">longest</span><span class="p">,</span> <span class="n">longest</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">buf</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">im</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="s2">&#34;WEBP&#34;</span><span class="p">,</span> <span class="n">quality</span><span class="o">=</span><span class="n">quality</span><span class="p">,</span> <span class="n">icc_profile</span><span class="o">=</span><span class="n">im</span><span class="o">.</span><span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;icc_profile&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">ContentFile</span><span class="p">(</span><span class="n">buf</span><span class="o">.</span><span class="n">getvalue</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">upload</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">Image</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">BytesIO</span><span class="p">(</span><span class="n">data</span><span class="p">))</span> <span class="k">as</span> <span class="n">im</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">im</span> <span class="o">=</span> <span class="n">ImageOps</span><span class="o">.</span><span class="n">exif_transpose</span><span class="p">(</span><span class="n">im</span><span class="p">)</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s2">&#34;RGB&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">im</span><span class="o">.</span><span class="n">thumbnail</span><span class="p">((</span><span class="mi">1080</span><span class="p">,</span> <span class="mi">1080</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">width</span><span class="p">,</span> <span class="n">upload</span><span class="o">.</span><span class="n">height</span> <span class="o">=</span> <span class="n">im</span><span class="o">.</span><span class="n">size</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">dominant_color</span> <span class="o">=</span> <span class="s2">&#34;#</span><span class="si">%02x%02x%02x</span><span class="s2">&#34;</span> <span class="o">%</span> <span class="n">im</span><span class="o">.</span><span class="n">resize</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span><span class="o">.</span><span class="n">getpixel</span><span class="p">((</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="si">}</span><span class="s2">.webp&#34;</span><span class="p">,</span> <span class="n">to_webp</span><span class="p">(</span><span class="n">im</span><span class="p">,</span> <span class="mi">1080</span><span class="p">,</span> <span class="mi">80</span><span class="p">),</span> <span class="n">save</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">thumbnail</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="si">}</span><span class="s2">_t.webp&#34;</span><span class="p">,</span> <span class="n">to_webp</span><span class="p">(</span><span class="n">im</span><span class="p">,</span> <span class="mi">320</span><span class="p">,</span> <span class="mi">70</span><span class="p">),</span> <span class="n">save</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">blurhash</span> <span class="o">=</span> <span class="n">blurhash</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">im</span><span class="o">.</span><span class="n">resize</span><span class="p">((</span><span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">)),</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span><span class="o">.</span><span class="n">processed_at</span> <span class="o">=</span> <span class="n">timezone</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span><span class="o">.</span><span class="n">save</span><span class="p">()</span></span></span></code></pre></div></div>
<p>The first line of <code>on_raw_photo</code> matters more than it looks. The router sends every event under <code>uploads/raw/</code> here, including the deletes from the orphan sweep and from the size check itself, and <code>get_object</code> on a key that&rsquo;s gone would only raise.</p>
<p>So the client&rsquo;s shrink was only ever for the pipe. The server still decides the format (WebP), the sizes, the thumbnail, the placeholder, and the dimensions it stores. Re-encoding through Pillow also drops the EXIF block, which is where phone cameras put the GPS position, so the privacy strip comes free. The ICC profile would go the same way, which shifts wide-gamut phone photos a touch, so <code>to_webp</code> passes it through. If you&rsquo;d rather keep pixel-perfect originals, don&rsquo;t shrink on the client; the flow is identical, the PUTs are just bigger, and bigger PUTs are precisely the thing this design makes cheap. One catch: originals from an iPhone are HEIC, which Pillow only opens with <code>pillow-heif</code> installed.</p>

<h3 class="relative group">4. The save attaches the photos and never waits on the webhook
    <div id="4-the-save-attaches-the-photos-and-never-waits-on-the-webhook" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#4-the-save-attaches-the-photos-and-never-waits-on-the-webhook" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When they hit Post, the request carries <code>upload_ids</code> instead of image data. The server looks each one up, with three outcomes:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">take</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">upload_ids</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">uploads</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">PhotoUpload</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span> <span class="n">public_id__in</span><span class="o">=</span><span class="n">upload_ids</span><span class="p">,</span> <span class="n">expires_at__gt</span><span class="o">=</span><span class="n">timezone</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">uploads</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">upload_ids</span><span class="p">)):</span>     <span class="c1"># expired, or not this user&#39;s</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="n">UploadNotReadyError</span><span class="p">(</span><span class="n">upload_ids</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">upload</span> <span class="ow">in</span> <span class="n">uploads</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">upload</span><span class="o">.</span><span class="n">processed_at</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>          <span class="c1"># the webhook hasn&#39;t got here yet</span>
</span></span><span class="line"><span class="cl">            <span class="n">data</span> <span class="o">=</span> <span class="n">read_raw</span><span class="p">(</span><span class="n">upload</span><span class="p">)</span>              <span class="c1"># None if the PUT never landed</span>
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">raise</span> <span class="n">UploadNotReadyError</span><span class="p">(</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">process</span><span class="p">(</span><span class="n">upload</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">uploads</span></span></span></code></pre></div></div>
<p>If the notification already ran, the processed files are attached as they are. If the file is in the bucket but the notification hasn&rsquo;t arrived, the save processes it inline, right there, so a post never waits on the webhook. The notification is a head start, not a dependency. And if the file isn&rsquo;t there at all, or an id has expired or belongs to someone else, the save returns a validation error the app retries after a moment; the app waits for its own PUTs before sending, so in practice that&rsquo;s rare.</p>
<p>A few details that made it hold up. The slot belongs to a user, so the server only accepts upload ids the same person created; otherwise anyone who saw an id could attach your photo to their listing. Everything falls back: if the slot request fails, the PUT is refused, or the API is an older version, the app sends base64 exactly as before. It only takes the upload path when every new photo has a slot, so a post&rsquo;s pictures are all uploads or all base64, and their order stays one list. And orphans expire: people pick photos and then give up, so an upload that&rsquo;s never attached expires after two days and the existing orphaned-media sweep cleans its files up.</p>

<h2 class="relative group">Presigned URL upload gotchas: CORS, size limits, expiry, multiple files
    <div id="presigned-url-upload-gotchas-cors-size-limits-expiry-multiple-files" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#presigned-url-upload-gotchas-cors-size-limits-expiry-multiple-files" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most of what goes wrong with presigned uploads goes wrong before the notification ever fires. These are the ones I&rsquo;d want on one page.</p>

<h3 class="relative group">CORS, for browsers only
    <div id="cors-for-browsers-only" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cors-for-browsers-only" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>A native app doesn&rsquo;t care. A browser sends a preflight <code>OPTIONS</code> to the bucket before the PUT, and the upload fails with a CORS error unless the bucket&rsquo;s CORS rules allow <code>PUT</code> from your origin with the <code>Content-Type</code> header. Tigris takes the standard S3 CORS configuration, so the rule looks the same as on AWS. It&rsquo;s the first thing to check when a presigned PUT works from <code>curl</code> and not from the page.</p>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="Tigris bucket settings, CORS and headers tab: one rule allowing PUT from two origins with the Content-Type header"
    width="1204"
    height="671"
    src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_7f61261034a8a56.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_7f61261034a8a56.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_3aa47360232f6475.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_3aa47360232f6475.webp"></span><figcaption>One CORS rule is all a browser upload needs: your origins, PUT, and the Content-Type header the signature expects.</figcaption></figure>
</p>

<h3 class="relative group">The content type is part of the signature
    <div id="the-content-type-is-part-of-the-signature" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-content-type-is-part-of-the-signature" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Signing with <code>ContentType</code> means the client must send exactly that header, or the bucket answers 403 <code>SignatureDoesNotMatch</code>. That&rsquo;s why the slot returns the headers along with the URL. Sign it on purpose, but know what it buys you: the header is enforced, the content isn&rsquo;t, so a PDF labelled <code>image/jpeg</code> gets through. The pipeline is the real check; Pillow refuses to open what isn&rsquo;t an image, and the row is never marked processed.</p>

<h3 class="relative group">Size limits
    <div id="size-limits" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#size-limits" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>A presigned PUT URL carries no size limit. If you need a hard stop before the bytes land, a presigned POST with a <code>content-length-range</code> policy condition does that. I didn&rsquo;t bother, because the notification carries the object&rsquo;s <code>size</code>: the handler deletes anything over the limit before reading it, and the save-time path reads at most the limit plus one byte so it can tell. The app sends a few hundred kilobytes; anything near fifteen megabytes isn&rsquo;t ours.</p>

<h3 class="relative group">Expiry
    <div id="expiry" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#expiry" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Two clocks. The presigned URL is good for fifteen minutes, which is plenty for one PUT and short enough that a leaked URL is useless soon. The slot row is good for two days, which covers someone picking photos, putting the phone down and finishing the listing tomorrow. The client treats a pending upload as stale after a day and starts a fresh one rather than hoping.</p>

<h3 class="relative group">Multiple files, and bigger ones
    <div id="multiple-files-and-bigger-ones" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#multiple-files-and-bigger-ones" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>One slot per file. Ten photos is ten slots and ten parallel PUTs, each with its own progress (an <code>XMLHttpRequest</code> gives you <code>upload.onprogress</code>; <code>fetch</code> doesn&rsquo;t), which you can&rsquo;t get from one giant base64 POST. Past a few megabytes, use multipart (parts are 5 MiB minimum, so it&rsquo;s for files well beyond that): create the multipart upload on the server, presign each <code>upload_part</code>, and complete it when the parts are in. Tigris sends <code>OBJECT_CREATED_MULTIPART</code> when the upload is completed, so the same handler fires and nothing else changes.</p>

<h3 class="relative group">Out of order
    <div id="out-of-order" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#out-of-order" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Events can arrive out of order, because the object may have been written in more than one region. For uploads it doesn&rsquo;t matter: one key, one event. For a key that gets overwritten, trust the eTag and the object&rsquo;s <code>Last-Modified</code>, not the order the webhooks reached you.</p>

<h2 class="relative group">The same signal for logs: alerts to Discord
    <div id="the-same-signal-for-logs-alerts-to-discord" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-same-signal-for-logs-alerts-to-discord" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Once the bucket can talk to the backend, anything that lands in it can be reacted to the same way, and the second thing I pointed it at was logs.</p>
<p>The app&rsquo;s logs are shipped to the bucket as small gzipped files, one every five minutes or so per app. For most of the year a GitHub Action on a 15-minute cron read the last 20 minutes of that archive with DuckDB, found the slow-request lines, and posted a digest to Discord. On paper that&rsquo;s fine. In practice GitHub&rsquo;s scheduled workflows don&rsquo;t run on time when the platform is busy, so &ldquo;every 15 minutes&rdquo; was more like &ldquo;every few hours&rdquo;, and when it finally ran it had a bigger window to chew through and occasionally ran out of memory. The overlap I&rsquo;d added so nothing slipped between runs meant the same slow request sometimes showed up twice. It was a lot of machinery for &ldquo;tell me when a request is slow&rdquo;.</p>
<p>Now each log file is read once, as it lands. They&rsquo;re a few kilobytes gzipped. The handler pulls the slow-request lines out, and if there are enough of them, or one really slow one, it posts a digest to a Discord channel through a webhook. Same for a burst of 5xx responses in one file. A Discord webhook from Python is one call:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">discord_alert</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">settings</span><span class="o">.</span><span class="n">DISCORD_WEBHOOK_URL</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="n">text</span><span class="p">[:</span><span class="mi">1900</span><span class="p">]},</span>  <span class="c1"># Discord caps a message at 2000 chars</span>
</span></span><span class="line"><span class="cl">        <span class="n">timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span></span></span></code></pre></div></div>
<p>No cron, no overlap, no duplicates, and the alert arrives within a few minutes of the request instead of whenever GitHub got round to it. The Action is gone.</p>
<p>While each file is open anyway, I also count what&rsquo;s in it (errors, warnings, health checks) into an hourly tally table. The daily summary used to scan a whole day of the archive. Now it reads a few hundred rows, and only falls back to the archive if the tally has a gap.</p>

<h2 class="relative group">A guard on deletes
    <div id="a-guard-on-deletes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-guard-on-deletes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This one I didn&rsquo;t plan. It fell out of having <code>OBJECT_DELETED</code> events. When a media file is deleted, the handler checks whether any row still points at it. If one does, that&rsquo;s a bug somewhere (or a person in the bucket console with a heavy finger), so it clears the column (the app shows a placeholder instead of a broken image) and reports it.</p>
<p>It also counts deletions per hour. Normal housekeeping removes a handful. If it ever sees a couple of hundred in an hour, I want to hear about that while it&rsquo;s happening, not when users start reporting blank photos.</p>

<h2 class="relative group">Things worth knowing
    <div id="things-worth-knowing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#things-worth-knowing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Respond fast. Tigris gives each delivery ten seconds. Reading a 5 KB log file or processing one phone photo fits; anything that might take longer goes to a queue and the view returns 200, or Tigris times out and retries, and you&rsquo;re back to deduplicating.</p>
<p>Filter at the bucket. My log shipper writes a file every few minutes per app, forever. If you only care about one prefix, say so in the notification&rsquo;s filter so your app isn&rsquo;t woken for everything else.</p>
<p>Test with a fake body. The payload is small and stable, so the tests post JSON at the view with the right <code>Authorization</code> header and assert what the handler did. No real storage involved.</p>
<p>Don&rsquo;t make it load-bearing on day one. Every use here has a fallback: the save can process a photo itself, the daily summary can still scan the archive, the app can still send base64. Notifications made each of them faster and cheaper, but none of them falls over if a webhook goes missing.</p>
<p>The best bit of the week was deleting that workflow file. Forty lines of YAML, a Python script, two repository secrets and a recurring &ldquo;why didn&rsquo;t the alert fire&rdquo; are gone, and the replacement is one handler that reads one small file when it arrives. The uploads were the point, though. The photos go up as fast as the network allows, the server still gets the last word on every one of them, and nobody had to build a bridge between the bucket and the backend, because the bucket now just says what happened.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
