<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>PostgreSQL · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/postgresql/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Tue, 17 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/postgresql/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Cloudflare 520, nginx 499 and Postgres Stalls: Keeping Webhooks Alive</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/</guid>
      <pubDate>Tue, 17 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>PostgreSQL</category>
      <category>PgBouncer</category>
      <category>Django</category>
      <category>Cloudflare</category>
      <category>nginx</category>
      <category>Webhooks</category>
      <category>Site Reliability</category>
      <category>Observability</category>
      <category>DevOps</category>
      <description>A 20-second Postgres stall timed out a chatbot&#39;s webhooks. How I held and retried them, and fixed Cloudflare 520 errors, nginx 499s and an exhausted PgBouncer pool.</description>
      <content:encoded><![CDATA[<p>The database behind a chatbot service I look after recently stalled twice, for about 20 seconds each time. Nothing crashed. Postgres came back by itself, the app came back with it, and the error rate barely moved. But the service&rsquo;s job is to take chat messages from a partner&rsquo;s system and acknowledge them quickly, and for those 20 seconds an acknowledgement that normally takes 0.4 s took 10 to 20 s. The partner&rsquo;s system doesn&rsquo;t wait that long. It gave up on those messages, and nothing on our side told anyone.</p>
<p>What caused them is still under investigation. The suspects so far are heavy queries, in particular ones that compare long text fields row by row, and several have been rewritten since. But none is confirmed as the cause yet.</p>
<p>So this post is about the other half of the problem, what the service does while the database isn&rsquo;t answering, and what I built after the stalls: a way to find out when the database stalls, a spool that holds requests on disk until it&rsquo;s back, a replay that runs each one exactly once, and a pass over every timeout between the internet and Postgres. A few days later it got a real test I hadn&rsquo;t planned.</p>
<p>If you landed here from one of these errors, each has its own section: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" >Cloudflare 520</a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" ><code>server closed the connection unexpectedly</code></a>, <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#nginx-499-client-closed-request-replies-nobody-received" >nginx 499</a> and <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/#pgbouncer-pool-exhausted-the-real-test" >a PgBouncer pool that ran dry</a>.</p>
<figure><div class="film-pair">
  <div class="mc-film film-dark">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRooAAABXRUJQVlA4IH4AAAAQBQCdASogABIAPw1urEqmIyIYDAQAYxDE9IAECBxAGNSFBxYLELF/yzl596pm6x2gAP7r1B4uhu6S+JqEfZ/87AtHGoj32Tp5yLTqb/5AlWmTYy/0oE8Nrih94UDfWuilEh2jC+eSMIqPA7MB5pMgkX94ddaOOys6I/gAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <div class="mc-film film-light">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light-poster.jpg" style="background:url('data:image/webp;base64,UklGRowAAABXRUJQVlA4IIAAAAAQBQCdASogABIAPw12qkmmI6GYDAQAYxDE9IAHOC3BZAFaEzG+kH6nxw8hG6nIiLJgAP7zbqngFyuj4/0DXJ2xZhiK6QUnRz4fsBZ+3lqB7T1fFerRg31UlZF+A/PCzrCTelVcbpcgNZFBnWzNRBEskzdVca60ciOypAYYRhAAAA==') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film-light.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/postgres-stall-film_en.vtt" srclang="en" label="English">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script></div><figcaption>This post as an eight-minute film, if you&rsquo;d rather watch than read. Some footage from Coverr.</figcaption>
</figure>


<h2 class="relative group">The requests
    <div id="the-requests" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-requests" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The partner&rsquo;s system sends each new chat message to a webhook endpoint. We answer <code>202 Accepted</code> with a job ID, a background worker generates the bot&rsquo;s reply, and the reply is posted back to a URL the partner gave us. Creating the job is one row in Postgres, so the 202 needs the database. Normally that takes about 0.4 s, and 2.6 s at the 99th percentile.</p>
<p>There&rsquo;s a second kind of client too: a browser extension that asks for a reply and waits for it in the same request, for up to 60 seconds.</p>
<p>The stack is ordinary. Cloudflare in front, nginx, Django, PgBouncer in transaction mode, and Postgres replicated across two machines in different places, with HAProxy sending connections to whichever one is the primary. The app server that takes this traffic runs next to the replica, so its writes cross the internet to the primary. That detail matters later.</p>

<h2 class="relative group">Detecting a Postgres stall before your users do
    <div id="detecting-a-postgres-stall-before-your-users-do" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#detecting-a-postgres-stall-before-your-users-do" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The first fix was the cheapest: know when it happens. The checks we had watched uptime, connection counts and locks, and a 20-second stall tripped none of them. Requests just took longer.</p>
<p>So I added a probe that runs on the host, outside the app, and does what the app does: it connects through the same proxy and commits a transaction, every two seconds. <code>txid_current()</code> makes Postgres assign a transaction ID, which makes it a write without touching any table. Normally it takes about 5 ms. Two commits slower than 2 s is a warning; one slower than 10 s, failing, or hanging is critical.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">psycopg</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">DSN</span> <span class="o">=</span> <span class="s2">&#34;postgresql://monitor@db-proxy:5432/app&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">commit_time</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">psycopg</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">DSN</span><span class="p">,</span> <span class="n">connect_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SET statement_timeout = &#39;15s&#39;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;SELECT txid_current()&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">took</span> <span class="o">=</span> <span class="n">commit_time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database commit failed: </span><span class="si">{</span><span class="n">exc</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">took</span> <span class="o">&gt;=</span> <span class="mi">10</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">alert</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;database stalled: a commit took </span><span class="si">{</span><span class="n">took</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> s&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The same script reads the web server&rsquo;s access log and checks how long the 202s took in the last minute: three over 5 s is a warning, any over 10 s is critical. That&rsquo;s the number the partner actually feels.</p>
<p>Two choices in it are deliberate. It runs on the host, not inside the app, so it still works when the app is the thing that&rsquo;s stuck. And it sends alerts straight to the chat channel, not through the app&rsquo;s own alerting, for the same reason. Each condition has a 15-minute cooldown and sends a message when it clears, so a flapping database doesn&rsquo;t flood the channel.</p>

<h2 class="relative group">Holding webhook requests on disk while the database is stuck
    <div id="holding-webhook-requests-on-disk-while-the-database-is-stuck" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#holding-webhook-requests-on-disk-while-the-database-is-stuck" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Knowing about a stall doesn&rsquo;t save the requests in it. For that, the API has to answer the partner within its patience even when the database can&rsquo;t.</p>
<p>Django middleware does it. It runs the real view in a thread and waits up to a budget of 4 seconds, which is well above the normal 99th percentile. If the view finishes in time, its response goes back as usual. If it doesn&rsquo;t, the middleware writes the raw request to a spool directory on disk and answers with a 202 of its own:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span> <span class="nt">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="nt">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">null</span><span class="p">,</span> <span class="nt">&#34;held&#34;</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span></span></span></code></pre></div></div>
<p>The view&rsquo;s thread keeps running, and if the database comes back quickly it finishes the job itself. The partner&rsquo;s documentation says to treat a held 202 exactly like a normal one and not to resend it.</p>
<p>Here&rsquo;s the shape of it, cut down:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SPOOL</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s2">&#34;/var/spool/myapp/held&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">BUDGET_SECONDS</span> <span class="o">=</span> <span class="mf">4.0</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">HoldOnStallMiddleware</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">get_response</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span> <span class="o">=</span> <span class="n">get_response</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">path</span> <span class="o">!=</span> <span class="s2">&#34;/api/jobs/&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">body</span>  <span class="c1"># read it before another thread can</span>
</span></span><span class="line"><span class="cl">        <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span><span class="o">.</span><span class="n">hex</span>
</span></span><span class="line"><span class="cl">        <span class="n">arrived</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">result</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">def</span> <span class="nf">run</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">worker</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">run</span><span class="p">,</span> <span class="n">daemon</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">worker</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BUDGET_SECONDS</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="s2">&#34;response&#34;</span> <span class="ow">in</span> <span class="n">result</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">result</span><span class="p">[</span><span class="s2">&#34;response&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">SPOOL</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">parents</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">held</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;key&#34;</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="p">,</span> <span class="s2">&#34;arrived&#34;</span><span class="p">:</span> <span class="n">arrived</span><span class="p">,</span> <span class="s2">&#34;body&#34;</span><span class="p">:</span> <span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">()}</span>
</span></span><span class="line"><span class="cl">        <span class="p">(</span><span class="n">SPOOL</span> <span class="o">/</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">arrived</span><span class="si">:</span><span class="s2">.6f</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="n">request</span><span class="o">.</span><span class="n">hold_key</span><span class="si">}</span><span class="s2">.json&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">write_text</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">held</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;pending&#34;</span><span class="p">,</span> <span class="s2">&#34;job_id&#34;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="s2">&#34;held&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">202</span><span class="p">)</span></span></span></code></pre></div></div>
<p>It sits ahead of every middleware that touches the database, including sessions and authentication, or those would stall first and the budget would never start.</p>

<h2 class="relative group">Retrying held webhooks exactly once
    <div id="retrying-held-webhooks-exactly-once" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#retrying-held-webhooks-exactly-once" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A separate process drains the spool. It waits until <code>SELECT 1</code> answers, then replays the held requests oldest first through the real endpoint, with a signed header that carries the hold key and the original arrival time. A 202 removes the file. A 429 or a 5xx means try again later, up to six hours. Anything else moves the file to a <code>failed/</code> folder for a person to look at. A lock file keeps it to one drainer per host, and it exits every hour so the next one always runs the current code.</p>
<p>The hard part is &ldquo;exactly once&rdquo;. When the database comes back, two things may try to create the same job: the original request thread, which was still waiting, and the drainer. The obvious answer is to hand the request over in memory, so the thread stops and leaves it to the drainer. A stall test in a sandbox showed why that doesn&rsquo;t work: the stall can start after the thread has already sent its <code>INSERT</code>. The thread can&rsquo;t take back a write it&rsquo;s waiting on.</p>
<p>So the database decides. Every job stores its hold key under a partial unique index, and whichever insert comes second fails:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">IntegrityError</span><span class="p">,</span> <span class="n">models</span><span class="p">,</span> <span class="n">transaction</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Job</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">hold_key</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">blank</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">payload</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">JSONField</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">constraints</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">            <span class="n">models</span><span class="o">.</span><span class="n">UniqueConstraint</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">fields</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;hold_key&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">                <span class="n">condition</span><span class="o">=</span><span class="n">models</span><span class="o">.</span><span class="n">Q</span><span class="p">(</span><span class="n">hold_key__isnull</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">                <span class="n">name</span><span class="o">=</span><span class="s2">&#34;job_hold_key_uniq&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_job</span><span class="p">(</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">with</span> <span class="n">transaction</span><span class="o">.</span><span class="n">atomic</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">IntegrityError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">Job</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">hold_key</span><span class="o">=</span><span class="n">hold_key</span><span class="p">)</span></span></span></code></pre></div></div>
<p>If creating a job also cancels an older one (ours does, when a newer request arrives for the same webhook), both statements go in one transaction, so the loser of the race cancels nothing.</p>
<p>Arrival time matters too. A held request is judged by when it arrived, not when it was replayed. It only replaces jobs created before it arrived, and if a newer request for the same webhook was queued while it sat on disk, it&rsquo;s dropped as superseded rather than run late.</p>

<h2 class="relative group">The first false alarm
    <div id="the-first-false-alarm" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-first-false-alarm" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Within hours of going live, the spool was holding three requests an hour, with no stall anywhere. Each was replayed within seconds, and each sent a warning and a summary to the ops channel, which buried anything real.</p>
<p>The cause was in front of the 202. For some requests the view described an image before creating the job, which took 1.6 to 2.1 s for an image it hadn&rsquo;t seen before, and occasionally pushed the acknowledgement past 4 s. The worker describes the image again anyway, so the early call only made the 202 slower. It moved into the worker, and the drainer now alerts only when a held request has waited 30 seconds without being replayed.</p>
<p>A spool with a short budget turns out to be a good way to find everything slow that sits in front of an acknowledgement.</p>

<h2 class="relative group">Finding the failing hop with Cloudflare analytics
    <div id="finding-the-failing-hop-with-cloudflare-analytics" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#finding-the-failing-hop-with-cloudflare-analytics" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Our own logs only see requests that reach the app. When something fails between the client and Django, the app has nothing to say about it. Cloudflare sees every request first, and it can tell you which side failed.</p>
<p>The zone is on the free plan, which has no Logpush, but the GraphQL Analytics API works on it. It gives request counts by edge status over time, percentiles of <code>originResponseDurationMs</code> (how long our side took to answer Cloudflare, including the proxy and the network), and sampled records of individual requests with both the status Cloudflare returned and the status our origin returned:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -s https://api.cloudflare.com/client/v4/graphql <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Authorization: Bearer </span><span class="nv">$CF_ANALYTICS_TOKEN</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  --data @- <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">{&#34;query&#34;: &#34;query($zone: String!, $since: Time!) { viewer { zones(filter: {zoneTag: $zone}) { httpRequestsAdaptive(limit: 50, orderBy: [datetime_DESC], filter: {datetime_geq: $since, edgeResponseStatus_geq: 499}) { datetime edgeResponseStatus originResponseStatus originResponseDurationMs clientRequestPath } } } }&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s"> &#34;variables&#34;: {&#34;zone&#34;: &#34;YOUR_ZONE_ID&#34;, &#34;since&#34;: &#34;YOUR_START_TIME&#34;}}
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span></span></span></code></pre></div></div>
<p><code>YOUR_START_TIME</code> is an ISO 8601 time in UTC. The Cloudflare analytics API token only needs the zone&rsquo;s Analytics Read permission. I put the results on an internal page next to the app&rsquo;s own timings.</p>
<p>Three things made it useful.</p>
<p><strong>Origin time against our own time.</strong> Cloudflare&rsquo;s origin time includes everything after the request leaves Cloudflare; the app&rsquo;s request traces cover only our code. The first reading was an origin median of 730 to 895 ms against about 500 ms of handling in the app, so a few hundred milliseconds were going somewhere before our code ran.</p>
<p><strong>The errors that never reach your logs.</strong> A 499 means the client hung up before we answered. 520 to 526 mean Cloudflare couldn&rsquo;t get a good answer from the origin. Neither shows up in Django. (A 504 from Cloudflare can be a different story again: <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >some aren&rsquo;t real</a>.)</p>
<p><strong>The <code>CF-Ray</code> header.</strong> Cloudflare adds it to every request it forwards. The app now stores it on each request&rsquo;s trace, so a request Cloudflare saw can be matched to one the app handled. An edge error with no matching trace never reached the app, which narrows the problem to the hops in between.</p>
<p>That last test found two problems.</p>

<h3 class="relative group">Cloudflare 520 error fix: nginx keepalive_timeout longer than Cloudflare&rsquo;s
    <div id="cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudflare-520-error-fix-nginx-keepalive_timeout-longer-than-cloudflares" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Cloudflare returned three 520s on the webhook endpoint one day, the error its page describes as &ldquo;Web server is returning an unknown error&rdquo;, and none of them had a trace in the app. So it was the proxy.</p>
<p>Cloudflare keeps idle connections to your origin open and reuses them for up to 900 seconds. nginx was closing idle keepalive connections after 90. Now and then Cloudflare sent a request down a connection at the moment nginx closed it, and got nothing back. The fix is one line:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="c1"># Longer than Cloudflare reuses an idle connection to the origin (900 s)
</span></span></span><span class="line"><span class="cl"><span class="k">keepalive_timeout</span> <span class="s">930s</span><span class="p">;</span></span></span></code></pre></div></div>

<h3 class="relative group">server closed the connection unexpectedly (Postgres): HAProxy and PgBouncer timeouts
    <div id="server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#server-closed-the-connection-unexpectedly-postgres-haproxy-and-pgbouncer-timeouts" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>It was the same kind of bug I&rsquo;d fixed lower down months before. HAProxy&rsquo;s 30-second timeouts were closing PgBouncer&rsquo;s idle connections to Postgres, and the app saw <code>server closed the connection unexpectedly</code>. HAProxy now allows 600 seconds, longer than PgBouncer keeps an idle server connection (300), and Django checks a reused connection before handing it out (<code>CONN_HEALTH_CHECKS = True</code>). The rule is the same at both layers: whoever holds an idle connection open must keep it longer than whoever reuses it.</p>

<h3 class="relative group">nginx 499 (client closed request): replies nobody received
    <div id="nginx-499-client-closed-request-replies-nobody-received" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#nginx-499-client-closed-request-replies-nobody-received" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When a client hangs up, Django still finishes the request and records a 200. Only nginx knows the client was gone, and it logs a 499. Logging those separately, with the request ID the app also uses, showed one office losing about 60% of its replies in the browser extension, to page reloads in the middle of a reply.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="k">map</span> <span class="nv">$status</span> <span class="nv">$client_gone</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kn">499</span>     <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">default</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">log_format</span> <span class="s">client_aborts</span> <span class="s">escape=json</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;</span><span class="p">{</span><span class="kn">&#34;time&#34;:&#34;$time_iso8601&#34;,&#34;request_id&#34;:&#34;$request_id&#34;,&#39;</span>
</span></span><span class="line"><span class="cl">    <span class="s">&#39;&#34;uri&#34;:&#34;</span><span class="nv">$uri&#34;,&#34;waited&#34;:$request_time}&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">access_log</span> <span class="s">/var/log/nginx/client-aborts.log</span> <span class="s">client_aborts</span> <span class="s">if=</span><span class="nv">$client_gone</span><span class="p">;</span></span></span></code></pre></div></div>
<p>A small job reads that file every 15 seconds and attaches each abort to its request trace once the request has finished, so a lost reply has a person and a conversation against it. An abort after 55 seconds or more is the extension&rsquo;s own 60-second timeout; anything shorter was usually a page reload.</p>
<p>One thing that doesn&rsquo;t fix a 499 is raising nginx&rsquo;s timeouts. The client is the one that gave up, so the fix is on the client&rsquo;s side or in answering sooner, and the useful part on the server is knowing which replies were lost.</p>

<h2 class="relative group">PgBouncer pool exhausted: the real test
    <div id="pgbouncer-pool-exhausted-the-real-test" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pgbouncer-pool-exhausted-the-real-test" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Then someone opened a statistics page. It sent one request per person on the team, 69 at once, and each pulled every row of that person&rsquo;s history through PgBouncer to the primary, across the internet. The pool has 50 connections plus 20 in reserve, and every one of them was taken.</p>
<p>For 1 minute 46 seconds, nothing that needed the database could get it. The spool held 80 webhook requests. All 80 were accounted for afterwards: 74 delivered, and 6 dropped because a newer request for the same webhook had replaced them. The browser extension has no spool, and 14 of its replies were lost.</p>
<p>So the spool worked, and the cause was ours. The fixes were about keeping heavy reads away from the path that serves requests:</p>
<ul>
<li><strong>One request instead of 69.</strong> The page asks for everyone at once, and the counts and averages are worked out in SQL instead of in Python over every row.</li>
<li><strong>Store what you&rsquo;ll want to average.</strong> The page scored each row by comparing two long text fields, in Python, for every row, which is why it pulled every row&rsquo;s full text across the internet. The score is now saved when the row is saved. For older rows, pairs that are identical count as a perfect score in SQL, and only the 2% or so that differ are fetched and compared. A timestamp a response-time query used to dig out of each conversation&rsquo;s history is saved when the row is created. That query used to scan up to 200 earlier messages per row; on a two-day window it went from 19.3 s to 1.5 s.</li>
<li><strong>Read from the node on the same machine.</strong> Reports get their own database connection to whichever Postgres node runs on the same host, read-only, with a statement timeout. If it&rsquo;s a replica more than five minutes behind, they use the normal connection instead.</li>
<li><strong>A cap.</strong> At most two report queries run at once per process; the rest wait up to 60 seconds and then get a 503.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># settings.py: a separate connection for heavy reads</span>
</span></span><span class="line"><span class="cl"><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;reports&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="o">**</span><span class="n">DATABASES</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;HOST&#34;</span><span class="p">:</span> <span class="s2">&#34;localhost&#34;</span><span class="p">,</span>  <span class="c1"># the Postgres node on this machine</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;CONN_MAX_AGE&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;OPTIONS&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;connect_timeout&#34;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;options&#34;</span><span class="p">:</span> <span class="s2">&#34;-c statement_timeout=120s -c default_transaction_read_only=on&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># reports.py: at most two heavy reads per process</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">_slots</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">BoundedSemaphore</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">heavy_read</span><span class="p">(</span><span class="n">view</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="nd">@wraps</span><span class="p">(</span><span class="n">view</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">_slots</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="s2">&#34;Reports are busy. Try again shortly.&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">_slots</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">wrapper</span></span></span></code></pre></div></div>
<p>One mistake in the first version is worth repeating. A failed report read was retried once on the normal connection, which made sense for a lost connection. But it also did it for a query that had hit its statement timeout, so the slowest query on the page ran for another 120 seconds on the primary, through the shared pool. Now only a lost connection falls back, a replica conflict is retried on the same node, and a timeout is reported as it is.</p>

<h2 class="relative group">Every timeout from Cloudflare to Postgres
    <div id="every-timeout-from-cloudflare-to-postgres" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-timeout-from-cloudflare-to-postgres" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most of what went wrong was two settings that only make sense together, set in different places by different people at different times. So here they all are, from the outside in:</p>
<table>
	<thead>
			<tr>
					<th>Hop</th>
					<th>Setting</th>
					<th>Value</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Partner&rsquo;s system</td>
					<td>Waits for our 202</td>
					<td>a few seconds (the exact limit is still under investigation)</td>
			</tr>
			<tr>
					<td>Our 202</td>
					<td>Hold budget before spooling</td>
					<td>4 s (normal p99 2.6 s)</td>
			</tr>
			<tr>
					<td>Browser extension</td>
					<td>Gives up on a reply</td>
					<td>60 s</td>
			</tr>
			<tr>
					<td>Cloudflare → nginx</td>
					<td>Reuses an idle connection</td>
					<td>up to 900 s</td>
			</tr>
			<tr>
					<td>nginx</td>
					<td><code>keepalive_timeout</code></td>
					<td>930 s</td>
			</tr>
			<tr>
					<td>Cloudflare</td>
					<td>Waits for the origin&rsquo;s response</td>
					<td>100 s (then a 524, &ldquo;A timeout occurred&rdquo;)</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td><code>timeout client</code> / <code>timeout server</code></td>
					<td>600 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>server_idle_timeout</code></td>
					<td>300 s</td>
			</tr>
			<tr>
					<td>PgBouncer</td>
					<td><code>query_wait_timeout</code> (waiting for a free connection)</td>
					<td>120 s</td>
			</tr>
			<tr>
					<td>Django</td>
					<td><code>CONN_MAX_AGE</code>, <code>CONN_HEALTH_CHECKS</code></td>
					<td>60 s, on</td>
			</tr>
			<tr>
					<td>Report connection</td>
					<td><code>statement_timeout</code>, <code>connect_timeout</code></td>
					<td>120 s, 3 s</td>
			</tr>
	</tbody>
</table>
<p>One pair in that table is still under investigation. If the report pages are served through Cloudflare&rsquo;s proxy, their 120-second statement timeout outlasts Cloudflare&rsquo;s 100 seconds, and a slow report would end in a 524 from Cloudflare before Postgres gives up on it.</p>
<p>Two rules cover nearly all of it. Idle timeouts should get longer as you go inwards: each hop keeps an idle connection open longer than the hop in front of it might reuse it. Waiting times should get shorter as you go inwards: each hop gives up, or answers something useful, before the one in front of it gives up on it. The 4-second hold budget is the second rule applied to the 202. Not every row follows it yet: PgBouncer will still queue a request for a free connection for 120 seconds, longer than either client waits.</p>

<h2 class="relative group">What I&rsquo;d do first next time
    <div id="what-id-do-first-next-time" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-do-first-next-time" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If I were setting this up again, I&rsquo;d start with the probe, because it costs an hour and tells you what you&rsquo;re dealing with. Then the Cloudflare analytics with <code>CF-Ray</code> on every trace, because it tells you which hop to look at. The spool came third, and it&rsquo;s the part I&rsquo;m happiest with, but it only made sense once I knew stalls were real and how long they lasted.</p>
<p>I&rsquo;d also take the browser extension&rsquo;s 14 lost replies as a hint. It has no spool because it waits for its answer in the same request, so there&rsquo;s nothing to hold. Giving it a job ID and a way to pick the answer up later would let it ride out the next stall the same way the webhooks did.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-520-nginx-499-postgres-stall/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Docker Compose in GitHub Actions CI: What I Replaced It With, and the Pipelines I Run Now</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-docker-compose-ci-github-actions/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-docker-compose-ci-github-actions/</guid>
      <pubDate>Thu, 22 Oct 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>CI/CD</category>
      <category>GitHub Actions</category>
      <category>Docker</category>
      <category>DevOps</category>
      <category>Deployment</category>
      <category>Fly.io</category>
      <category>Cloudflare</category>
      <category>Django</category>
      <category>PostgreSQL</category>
      <category>Hugo</category>
      <category>Python</category>
      <description>The GitHub Actions pipelines I run in 2026: service containers instead of Docker Compose in CI, caching chosen from step timings, and deploys that are scripts.</description>
      <content:encoded><![CDATA[<p>A CI/CD pipeline is the automated route a change takes from <code>git push</code> to production: CI builds and tests every push on a clean machine, and CD releases whatever passed through one repeatable step. On GitHub Actions that&rsquo;s a YAML file under <code>.github/workflows/</code>, and mine are deliberately thin: the checks run the commands I run on my laptop, and every deploy is a script or a CLI that would work the same without GitHub.</p>
<p>If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as <code>services:</code> and runs the same <code>uv run …</code> and <code>npm run …</code> commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.</p>
<p>In 2024 I wrote <a href="https://jared.lynskey.co.nz/en/posts/2024/2024-06-02-simplifyingcicd/" >Use Docker Compose for CI/CD Instead of Platform-Specific YAML</a>, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls <code>docker compose</code> any more. The principle held; the mechanism didn&rsquo;t. Here&rsquo;s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.</p>

<h2 class="relative group">Every pipeline I run, in one table
    <div id="every-pipeline-i-run-in-one-table" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-pipeline-i-run-in-one-table" aria-label="Anchor">#</a>
    </span>
    
</h2>
<table>
	<thead>
			<tr>
					<th>Pipeline</th>
					<th>Started by</th>
					<th>What CI checks</th>
					<th>Where it goes</th>
					<th>Who releases</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>This blog, and a client&rsquo;s Hugo site</td>
					<td>Every push, on any branch</td>
					<td>That <code>hugo --gc</code> builds</td>
					<td>Cloudflare Pages: <code>main</code> is production, other branches get a preview URL</td>
					<td>Nobody; the push is the release</td>
			</tr>
			<tr>
					<td>My marketplace app&rsquo;s API</td>
					<td>A deploy script, run by hand</td>
					<td>Lint, type checks and the Django tests, on every push to <code>main</code></td>
					<td>Fly.io, as a rolling deploy</td>
					<td>Me</td>
			</tr>
			<tr>
					<td>The same app&rsquo;s front end: web, OTA updates, store builds</td>
					<td>Release scripts and local store builds, by hand</td>
					<td>TypeScript and Jest, on every push to <code>main</code></td>
					<td>Object storage, an OTA update service and the two app stores</td>
					<td>Me</td>
			</tr>
			<tr>
					<td>pyhtml2text, my open-source Python package</td>
					<td>Every push, pull request and tag</td>
					<td>Wheels for 15 build targets, tested on most of them</td>
					<td>PyPI, on a tag</td>
					<td>The tag</td>
			</tr>
	</tbody>
</table>
<p>In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don&rsquo;t stop me deploying. That&rsquo;s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.</p>

<h2 class="relative group">The CI half: one workflow, two jobs
    <div id="the-ci-half-one-workflow-two-jobs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-ci-half-one-workflow-two-jobs" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The app&rsquo;s CI is one file, <code>.github/workflows/checks.yml</code>. Until 11 October there wasn&rsquo;t any. This is a trimmed example of its shape:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># Lint, types and tests on every push and pull request to main.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># Nothing here deploys.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Checks</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">push</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">main]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">pull_request</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">main]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">concurrency</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">checks-${{ github.ref }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cancel-in-progress</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">python</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">timeout-minutes</span><span class="p">:</span><span class="w"> </span><span class="m">30</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">postgres</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">postgres:17</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;5432:5432&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">options</span><span class="p">:</span><span class="w"> </span><span class="p">&gt;-</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          --health-cmd &#34;pg_isready -U myapp&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          --health-interval 5s --health-timeout 5s --health-retries 20</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">valkey</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">valkey/valkey:8-alpine</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;6379:6379&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">DEBUG</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;True&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">DATABASE_URL</span><span class="p">:</span><span class="w"> </span><span class="l">postgresql://myapp:myapp@localhost:5432/myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">REDIS_URL</span><span class="p">:</span><span class="w"> </span><span class="l">redis://localhost:6379</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">PYTHON_JIT</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;0&#34;</span><span class="w">   </span><span class="c"># match production</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">astral-sh/setup-uv@v10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">enable-cache</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">prune-cache</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv sync --locked</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv run ruff check .</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv run mypy myapp/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv run python manage.py test --parallel 4 --noinput</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">app</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">timeout-minutes</span><span class="p">:</span><span class="w"> </span><span class="m">20</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">defaults</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">run</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">working-directory</span><span class="p">:</span><span class="w"> </span><span class="l">frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-node@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">node-version</span><span class="p">:</span><span class="w"> </span><span class="m">24</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">cache</span><span class="p">:</span><span class="w"> </span><span class="l">npm</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">cache-dependency-path</span><span class="p">:</span><span class="w"> </span><span class="l">frontend/package-lock.json</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm ci</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Expo writes this on `expo start` and it is gitignored,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># so a fresh checkout doesn&#39;t have it.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">printf &#39;/// &lt;reference types=&#34;expo/types&#34; /&gt;\n&#39; &gt; expo-env.d.ts</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm run typecheck</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npx jest --ci</span></span></span></code></pre></div></div>
<p>The two jobs run side by side. Across the 35 green runs so far, the app job&rsquo;s median is about seven minutes and the Python job&rsquo;s about twelve and a half.</p>
<p>The <code>services:</code> block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default <code>docker compose up</code> gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.</p>
<p>CI runs the same interpreter as my laptop and the production image, uv&rsquo;s own build of Python 3.15.0, because <code>uv sync</code> reads <code>.python-version</code> and downloads exactly that build. (My real file also passes setup-uv a <code>python-version-file</code> input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)</p>
<p>Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15&rsquo;s JIT compiler, and the first run with it failed a test: a plain <code>yield from</code>, on a code path the app takes on every save, raised <code>AttributeError: 'list' object has no attribute 'send'</code>. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It&rsquo;s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.</p>
<p>The very first run failed too, for a duller reason. Expo writes <code>expo-env.d.ts</code> when you run <code>expo start</code>, and the file is gitignored, so every working copy I&rsquo;d used had one and a fresh checkout didn&rsquo;t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.</p>
<p><code>concurrency</code> cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.</p>

<h2 class="relative group">Where the Docker Compose file went
    <div id="where-the-docker-compose-file-went" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-the-docker-compose-file-went" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Here is the part of the app&rsquo;s <code>docker-compose.yml</code> that CI cares about, as an example. The real one has more services and a lot more environment.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">valkey</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">valkey/valkey:8-alpine</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;6379:6379&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="c"># Host port 55432, not 5432, so it can&#39;t collide with a local Postgres.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">postgres</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">postgres:17-alpine</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">POSTGRES_USER=myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">POSTGRES_PASSWORD=myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">POSTGRES_DB=myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;55432:5432&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">postgres_data:/var/lib/postgresql/data</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">healthcheck</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">test</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;CMD-SHELL&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;pg_isready -U myapp -d myapp&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">interval</span><span class="p">:</span><span class="w"> </span><span class="l">5s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">retries</span><span class="p">:</span><span class="w"> </span><span class="m">10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">api</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">build</span><span class="p">:</span><span class="w"> </span><span class="l">.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l">python manage.py runserver 0.0.0.0:8000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">.:/app</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;8000:8000&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">REDIS_URL=redis://valkey:6379</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">depends_on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">postgres</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">condition</span><span class="p">:</span><span class="w"> </span><span class="l">service_healthy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">valkey</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">condition</span><span class="p">:</span><span class="w"> </span><span class="l">service_started</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="l">postgres_data:</span></span></span></code></pre></div></div>
<p>Put it next to the <code>services:</code> block in <code>checks.yml</code> and it&rsquo;s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and <code>valkey/valkey:8-alpine</code>. That&rsquo;s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, <code>uv run …</code> and <code>npm run …</code>, straight on the runner.</p>
<p>Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn&rsquo;t survive is CI calling Compose.</p>
<p>The app&rsquo;s CI doesn&rsquo;t build the Docker image at all; Fly&rsquo;s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn&rsquo;t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.</p>
<p>And the 2024 example had a bug I didn&rsquo;t notice at the time. <code>docker compose build</code> builds the image and never runs the service&rsquo;s <code>command</code>, so the <code>npm test</code> in that file never ran in CI. The workflow went green whether the tests passed or not.</p>
<p>The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client&rsquo;s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text&rsquo;s. It&rsquo;s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.</p>

<h2 class="relative group">Caching: read the step timings first
    <div id="caching-read-the-step-timings-first" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#caching-read-the-step-timings-first" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>GitHub keeps the duration of every step, and <code>gh api repos/&lt;owner&gt;/&lt;repo&gt;/actions/runs/&lt;run-id&gt;/jobs</code> hands it over. These are from the app&rsquo;s latest green run, on 20 October:</p>
<table>
	<thead>
			<tr>
					<th>Job</th>
					<th>Step</th>
					<th>Seconds</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Python</td>
					<td>Start Postgres and Valkey</td>
					<td>18</td>
			</tr>
			<tr>
					<td>Python</td>
					<td><code>uv sync --locked</code>, cache warm</td>
					<td>6</td>
			</tr>
			<tr>
					<td>Python</td>
					<td>mypy</td>
					<td>27</td>
			</tr>
			<tr>
					<td>Python</td>
					<td>Django tests</td>
					<td>761</td>
			</tr>
			<tr>
					<td>App</td>
					<td><code>npm ci</code>, cache warm</td>
					<td>60</td>
			</tr>
			<tr>
					<td>App</td>
					<td>Jest</td>
					<td>333</td>
			</tr>
	</tbody>
</table>
<p>Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I&rsquo;d change the suite, not the cache.</p>
<p>The blog is the opposite case. Hugo&rsquo;s processed images live in the gitignored <code>resources/_gen</code>, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with <code>actions/cache</code>, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what&rsquo;s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.</p>
<p>The app&rsquo;s image builds on Fly&rsquo;s builder, with the dependency files copied before the code and uv&rsquo;s download cache in a BuildKit cache mount (<code>RUN --mount=type=cache,target=/root/.cache/uv</code>). The bigger fix wasn&rsquo;t a cache. The old <code>.dockerignore</code> was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"># Exclude everything, then let back in only what the Dockerfile COPYs.
</span></span><span class="line"><span class="cl">*
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">!pyproject.toml
</span></span><span class="line"><span class="cl">!uv.lock
</span></span><span class="line"><span class="cl">!manage.py
</span></span><span class="line"><span class="cl">!myapp
</span></span><span class="line"><span class="cl">!static
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"># Rules match the whole relative path and the last match wins,
</span></span><span class="line"><span class="cl"># so these re-excludes apply inside the allowed directories too.
</span></span><span class="line"><span class="cl">**/__pycache__/
</span></span><span class="line"><span class="cl">**/*.pyc
</span></span><span class="line"><span class="cl">**/.DS_Store
</span></span><span class="line"><span class="cl">**/*.log</span></span></code></pre></div></div>

<h2 class="relative group">Secrets, in as few places as possible
    <div id="secrets-in-as-few-places-as-possible" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#secrets-in-as-few-places-as-possible" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, <code>CLOUDFLARE_API_TOKEN</code>. Fly secrets are the app&rsquo;s runtime configuration, set with <code>flyctl secrets set</code>, and they never pass through Actions. And a gitignored <code>.env</code> on my laptop holds the token the deploy script uses to purge Cloudflare&rsquo;s cache.</p>
<p>PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for <code>id-token: write</code> and the repository&rsquo;s secrets page is empty. There&rsquo;s nothing to leak and nothing to rotate.</p>
<p>Tokens get the least they need. The app&rsquo;s Fly token in GitHub serves a scheduled check that only lists machines, so it&rsquo;s created with <code>fly tokens create readonly</code>. The Pages deploys ask for <code>contents: read</code> and <code>deployments: write</code>, and nothing more.</p>
<p>Inputs go through <code>env:</code>, never straight into a script. The OTA workflow takes free-text release notes. Written as <code>${{ inputs.notes }}</code> inside <code>run:</code>, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Publish</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">OTA_API_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.OTA_API_TOKEN }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">INPUT_PLATFORM</span><span class="p">:</span><span class="w"> </span><span class="l">${{ inputs.platform }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">INPUT_ROLLOUT</span><span class="p">:</span><span class="w"> </span><span class="l">${{ inputs.rollout }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">INPUT_NOTES</span><span class="p">:</span><span class="w"> </span><span class="l">${{ inputs.notes }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          args=(&#34;$INPUT_PLATFORM&#34;)
</span></span></span><span class="line"><span class="cl"><span class="sd">          [ -n &#34;$INPUT_ROLLOUT&#34; ] &amp;&amp; args+=(--rollout &#34;$INPUT_ROLLOUT&#34;)
</span></span></span><span class="line"><span class="cl"><span class="sd">          [ -n &#34;$INPUT_NOTES&#34; ] &amp;&amp; args+=(--notes &#34;$INPUT_NOTES&#34;)
</span></span></span><span class="line"><span class="cl"><span class="sd">          ./scripts/publish-ota.sh &#34;${args[@]}&#34;</span></span></span></code></pre></div></div>
<p>Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app&rsquo;s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.</p>

<h2 class="relative group">Deploys: on every push, by script, and on a tag
    <div id="deploys-on-every-push-by-script-and-on-a-tag" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#deploys-on-every-push-by-script-and-on-a-tag" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">The Hugo sites deploy on every push
    <div id="the-hugo-sites-deploy-on-every-push" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-hugo-sites-deploy-on-every-push" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>An example version of the blog&rsquo;s workflow, with the first-run step that creates the Pages project left out:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Deploy to Cloudflare Pages</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">push</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">workflow_dispatch</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">HUGO_VERSION</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;0.165.0&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">PAGES_PROJECT</span><span class="p">:</span><span class="w"> </span><span class="l">my-site</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">concurrency</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">deploy-${{ github.ref }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cancel-in-progress</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">deploy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">permissions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span><span class="l">read</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">deployments</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">submodules</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">peaceiris/actions-hugo@v3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">hugo-version</span><span class="p">:</span><span class="w"> </span><span class="l">${{ env.HUGO_VERSION }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">extended</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Processed images are gitignored; restore them so Hugo only</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># re-encodes what is new. An older cache still helps.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/cache@v6</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">resources/_gen</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">hugo-resources-${{ hashFiles(&#39;content/**/*.jpg&#39;, &#39;content/**/*.png&#39;, &#39;assets/img/**&#39;) }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">restore-keys</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">            hugo-resources-</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># A multilingual site has no root 404.html; without one, Pages</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># answers unknown paths with index.html and a 200.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          hugo --gc
</span></span></span><span class="line"><span class="cl"><span class="sd">          cp public/en/404.html public/404.html</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Deploy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">cloudflare/wrangler-action@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">apiToken</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.CLOUDFLARE_API_TOKEN }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">accountId</span><span class="p">:</span><span class="w"> </span><span class="l">${{ vars.CLOUDFLARE_ACCOUNT_ID }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l">pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}</span></span></span></code></pre></div></div>
<p><code>--branch=${{ github.ref_name }}</code> makes <code>main</code> production and gives every other branch a preview URL. The client site&rsquo;s workflow mirrors this one, plus an <code>rm -rf public</code> before the build, because its <code>public/</code> is committed and removed pages would otherwise linger in the upload.</p>
<p>Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.</p>
<p>And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo&rsquo;s <code>ref</code>, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with <code>REF_NOT_FOUND</code>. <code>main</code> carried on deploying in the meantime.</p>

<h3 class="relative group">The app&rsquo;s API deploys from a script
    <div id="the-apps-api-deploys-from-a-script" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-apps-api-deploys-from-a-script" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The core of the deploy script, as an example:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">IMAGE_LABEL</span><span class="o">=</span><span class="s2">&#34;deploy-</span><span class="k">$(</span>git rev-parse --short HEAD<span class="k">)</span><span class="s2">-</span><span class="k">$(</span>date +%Y%m%d%H%M%S<span class="k">)</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Build first, then wake a spare machine, then roll out exactly that image.</span>
</span></span><span class="line"><span class="cl">flyctl deploy --config fly.toml --build-only --push --image-label <span class="s2">&#34;</span><span class="nv">$IMAGE_LABEL</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">curl -fsS --retry <span class="m">5</span> https://myapp.fly.dev/health &gt; /dev/null
</span></span><span class="line"><span class="cl">flyctl deploy --config fly.toml --image <span class="s2">&#34;registry.fly.io/myapp:</span><span class="nv">$IMAGE_LABEL</span><span class="s2">&#34;</span></span></span></code></pre></div></div>
<p>And the matching lines in <code>fly.toml</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">deploy</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="c"># max_unavailable only counts machines that are running,</span>
</span></span><span class="line"><span class="cl">  <span class="c"># which is why the script wakes one first.</span>
</span></span><span class="line"><span class="cl">  <span class="nx">strategy</span> <span class="p">=</span> <span class="s2">&#34;rolling&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">max_unavailable</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p>The image is built once, pushed to Fly&rsquo;s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.</p>
<p>Waking a spare first is what makes it zero-downtime. <code>max_unavailable = 1</code> only counts running machines, and the app keeps none running when it&rsquo;s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain <code>flyctl deploy</code>, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.</p>
<p>The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare&rsquo;s cache and warms the machines. The web front end skips the image entirely; that&rsquo;s in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-14-web-releases-without-redeploying/" >Deploy Frontend and Backend Separately, Without Redeploying the API</a>.</p>
<p>Migrations aren&rsquo;t a pipeline step. Fly&rsquo;s <code>release_command</code> hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that&rsquo;s still running: add a column in one deploy, stop using it in the next, drop it in a third.</p>
<p>Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn&rsquo;t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.</p>
<p>In <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/" >Preview Environments per Branch on Fly.io</a> I wrote that doing this by hand is fine while you&rsquo;re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.</p>
<p>The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it&rsquo;s the same script.</p>

<h3 class="relative group">pyhtml2text releases on a tag
    <div id="pyhtml2text-releases-on-a-tag" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pyhtml2text-releases-on-a-tag" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The release job, with the package name made generic:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w">  </span><span class="nt">release</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">${{ startsWith(github.ref, &#39;refs/tags/&#39;) || github.event_name == &#39;workflow_dispatch&#39; }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">needs</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">linux, macos, windows, sdist]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">pypi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">https://pypi.org/p/mypackage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">permissions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">id-token</span><span class="p">:</span><span class="w"> </span><span class="l">write      </span><span class="w"> </span><span class="c"># trusted publishing and signing</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">attestations</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/download-artifact@v8</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/attest@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">subject-path</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;wheels-*/*&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">astral-sh/setup-uv@v10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">${{ startsWith(github.ref, &#39;refs/tags/&#39;) }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">prune-cache</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Publish to PyPI</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">${{ startsWith(github.ref, &#39;refs/tags/&#39;) }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv publish --trusted-publishing always &#39;wheels-*/*&#39;</span></span></span></code></pre></div></div>
<p>The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. <code>actions/attest</code> records which workflow built each wheel, and <code>uv publish --trusted-publishing always</code> uploads them with the OIDC token rather than a password. The package itself is in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-18-pyhtml2text/" >A Closed Pull Request, a Fork and a PyPI Package</a>.</p>

<h2 class="relative group">Scheduled workflows are best effort
    <div id="scheduled-workflows-are-best-effort" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#scheduled-workflows-are-best-effort" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The app has one scheduled workflow, a check that Fly hasn&rsquo;t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.</p>
<p>It&rsquo;s set to <code>cron: '*/15 * * * *'</code>, 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub&rsquo;s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it&rsquo;s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn&rsquo;t a monitor.</p>

<h2 class="relative group">What I&rsquo;d ask of a pipeline
    <div id="what-id-ask-of-a-pipeline" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-ask-of-a-pipeline" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you&rsquo;re building one, or looking at one you already have:</p>
<ol>
<li>If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.</li>
<li>Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn&rsquo;t.</li>
<li>Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn&rsquo;t.</li>
<li>Where does a run&rsquo;s time actually go? Read the step timings before you add a cache.</li>
<li>For each secret, how many places does it live, and does it need to be in GitHub at all?</li>
</ol>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-docker-compose-ci-github-actions/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Preview Environments per Branch on Fly.io: Ending the Fight for Staging</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/</guid>
      <pubDate>Sat, 19 Sep 2026 00:00:00 +1200</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Fly.io</category>
      <category>Cloudflare</category>
      <category>DevOps</category>
      <category>CI/CD</category>
      <category>GitHub Actions</category>
      <category>PostgreSQL</category>
      <category>Redis</category>
      <category>Django</category>
      <category>Database Migrations</category>
      <category>Branching</category>
      <category>Preview Environments</category>
      <description>How to give every branch a preview environment on Fly.io and Cloudflare: one PR label, shared Postgres, Redis and bucket sliced by a branch hash, gone when the PR closes.</description>
      <content:encoded><![CDATA[<p>To give every branch its own preview environment on Fly.io, I don&rsquo;t copy the infrastructure, I slice it. One <code>env</code> label on the PR starts a GitHub Actions workflow that runs one script; the script derives a name from the branch (a slug plus a short hash) and stamps it on a Fly app, a Postgres database cloned from a template, a Redis key prefix, a bucket prefix, a generated set of secrets and a hostname under a Cloudflare wildcard. Every push updates that environment, closing or merging the PR tears it down, and a nightly sweeper catches whatever leaks. They&rsquo;re ephemeral environments on shared infrastructure, a couple of minutes from label to URL, and nobody has to ask who&rsquo;s on staging.</p>
<p>Picture a small team working on one app. Two developers, or five; not specialists, but people who each take a feature from the database migration through the API to the screen in the mobile app, end to end, on their own branch. It&rsquo;s a good way to run a team, and from the second developer onward it has one recurring failure: the message &ldquo;Is anyone on staging?&rdquo;</p>
<p>Behind that message is a developer who has finished a feature and wants a tester or a designer to see it, and can&rsquo;t, because staging is running the second developer&rsquo;s branch, and dev is running the third&rsquo;s, which broke login two days ago and hasn&rsquo;t been redeployed since. A handful of people, two shared environments, and a chat thread doing the scheduling.</p>
<p>That isn&rsquo;t a people problem. It&rsquo;s a topology problem. Every developer is treating the environment as the place where their branch becomes real, so the environment is doing the job the merge is supposed to do, one branch at a time, and everyone else is queueing for it.</p>
<p>I described the production shape of my stack in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/" >My Current Stack</a>: Django on Fly.io&rsquo;s cheapest machines, Postgres behind a connection pooler, Redis for the cache and the Channels layer, a separate WebSocket app, a background worker, an S3-compatible bucket for files, a hosted auth provider, Cloudflare in front, and a mobile app talking to all of it. This post is how I turned that stack into something where any branch gets its own public, HTTPS-fronted copy of the whole thing a couple of minutes after its developer puts one label on the PR, with nobody else asked and nobody provisioning it, and without a second Postgres cluster, a second Redis or a second bucket. The infrastructure stays shared. What changes is that every shared resource learns to be sliced by a name, and the name comes from the branch. Nothing in it assumes a team size: it&rsquo;s the same design for two developers as for twenty, and what grows is the app list, which the sweeper handles. I built it, ran three branches at once, broke it in two places I hadn&rsquo;t predicted, and tore it down, all in an afternoon; the outputs below are from that run.</p>

<h2 class="relative group">Why one shared environment can&rsquo;t scale past one developer
    <div id="why-one-shared-environment-cant-scale-past-one-developer" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-one-shared-environment-cant-scale-past-one-developer" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A shared dev environment has an assumption baked in: that there is one &ldquo;current version of the code&rdquo; that everyone wants to look at. That&rsquo;s true for one developer, or for a team working a single stream of changes in lockstep. It stops being true the moment the second developer starts a feature on a different timeline, and with end-to-end ownership that&rsquo;s every feature. From then on, everything the environment holds is contended:</p>
<ul>
<li><strong>The deployed code.</strong> Whoever deployed last wins. Everyone else&rsquo;s work is invisible until they redeploy, which undoes the first person&rsquo;s.</li>
<li><strong>The database schema.</strong> This is the one that bites hardest when every developer owns their feature&rsquo;s data model. The first branch adds a column. The second, deployed an hour later, doesn&rsquo;t know about it and its ORM starts failing on inserts. Or both branches have a <code>0042_*</code> migration, the shared database ends up in a state that matches nobody&rsquo;s migration history, and someone resets it, deleting the test data the third developer spent an afternoon setting up.</li>
<li><strong>The cache.</strong> One branch&rsquo;s serialiser change is cached under the same Redis keys another branch&rsquo;s code reads. The bug reports that produces are baffling and cost a morning each.</li>
<li><strong>Third-party integrations.</strong> One OAuth callback, one webhook URL, one push certificate. Whoever&rsquo;s branch is deployed gets the events.</li>
<li><strong>The tester.</strong> They can only test what&rsquo;s deployed, so they test one feature at a time in the order the environment happened to be claimed, not the order the features are ready.</li>
</ul>
<p>The team responds with a booking system: a chat thread, a pinned message, a bot with <code>/claim staging</code>. That&rsquo;s a lock, and a lock on the only place your work can be seen is a lock on delivery rate. With two developers it&rsquo;s an irritation. With three it&rsquo;s tolerable on a good week. Add a fourth, or have one feature hold staging for three days of back-and-forth with a designer, and it stops being tolerable.</p>
<p>The first fix people reach for is a third shared environment, &ldquo;dev2&rdquo; or &ldquo;uat&rdquo;. That buys a few weeks. The queue just has two lanes.</p>
<p>The real fix is to notice that dev and staging were never environments. They were <em>the current state of someone&rsquo;s branch, with a URL</em>. If that&rsquo;s what they are, the right number of them is the number of branches the team cares about right now, and the right lifetime is the lifetime of the branch.</p>

<h2 class="relative group">The model: the branch is the environment
    <div id="the-model-the-branch-is-the-environment" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-model-the-branch-is-the-environment" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Here&rsquo;s the target state.</p>
<table>
	<thead>
			<tr>
					<th>Environment</th>
					<th>How many</th>
					<th>Created by</th>
					<th>Destroyed by</th>
					<th>Lives for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Branch environment (was &ldquo;dev&rdquo;)</td>
					<td>One per active branch</td>
					<td>An <code>env</code> label on the branch&rsquo;s PR, then every push; by hand, the same <code>./scripts/env up</code> for a spike without a PR</td>
					<td>Branch deletion, plus a nightly sweeper</td>
					<td>Days</td>
			</tr>
			<tr>
					<td>Staging</td>
					<td>One per release candidate</td>
					<td>CI when a tag is cut from <code>main</code></td>
					<td>CI once the same image is promoted to prod</td>
					<td>Hours</td>
			</tr>
			<tr>
					<td>Prod</td>
					<td>One</td>
					<td>You, once</td>
					<td>Nobody</td>
					<td>Forever</td>
			</tr>
	</tbody>
</table>
<p>Read the <em>Created by</em> and <em>Destroyed by</em> columns again: every entry is a git event or a clock. Git is the control plane. The set of environments that should exist is a function of the set of branches that exist, and the automation&rsquo;s only job is to keep reality matching it: a label on the PR creates, every push after that updates, closing or merging the PR destroys, and a nightly job reconciles whatever the first three missed. Nor does every branch get one: the label is the gate, one click by the developer, so a backup push or a bot&rsquo;s dependency bump costs nothing. Nobody provisions an environment. Not a DevOps engineer with a ticket, not a bot you ask in chat, not an AI agent you prompt each time. The tenth developer&rsquo;s first branch gets an environment the same way the first developer&rsquo;s did, by labelling its PR, and if a person has to be asked, it hasn&rsquo;t scaled; the queue has just moved.</p>
<p>And the rule that makes it work: <strong>every resource an environment touches is named after the branch, deterministically.</strong> Not after the developer, not after a ticket, not after a PR number that doesn&rsquo;t exist until someone opens one. After the branch, because the branch is what developers actually work with, and because a deterministic name means the second run of <code>./scripts/env up</code> on the same branch finds the environment it already made rather than making another.</p>
<p>The name is a slug for humans and a short hash for uniqueness:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">branch</span><span class="o">=</span><span class="k">$(</span>git rev-parse --abbrev-ref HEAD<span class="k">)</span>              <span class="c1"># feature/payments-retry</span>
</span></span><span class="line"><span class="cl"><span class="nv">slug</span><span class="o">=</span><span class="k">$(</span><span class="nb">printf</span> <span class="s1">&#39;%s&#39;</span> <span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> <span class="p">|</span> tr <span class="s1">&#39;[:upper:]/_ &#39;</span> <span class="s1">&#39;[:lower:]---&#39;</span> <span class="p">|</span> tr -cd <span class="s1">&#39;a-z0-9-&#39;</span> <span class="p">|</span> cut -c1-20<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">hash</span><span class="o">=</span><span class="k">$(</span><span class="nb">printf</span> <span class="s1">&#39;%s&#39;</span> <span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> <span class="p">|</span> sha256sum <span class="p">|</span> cut -c1-6<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">ENV</span><span class="o">=</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">slug</span><span class="p">%-</span><span class="si">}</span><span class="s2">-</span><span class="si">${</span><span class="nv">hash</span><span class="si">}</span><span class="s2">&#34;</span>                                 <span class="c1"># feature-payments-retr-3f9a1c</span></span></span></code></pre></div></div>
<p><code>feature-payments-retr-3f9a1c</code> is the environment. That string becomes the Fly app name, the Postgres database name (with hyphens turned to underscores), the Redis key prefix, the bucket&rsquo;s object prefix, the error tracker&rsquo;s environment, and the left-hand label of the hostname. Anyone who knows the branch name can compute it. Nothing has to be looked up.</p>
<p>That last sentence is the design. There is no registry of environments and no coordinator. <code>env up</code>, <code>env down</code>, the Cloudflare Worker that routes hostnames and the nightly sweeper each recompute the name from the branch independently and agree by construction. Cloudflare in particular is not coordinating anything: the Worker is a pure function from hostname to Fly app, and it never learns that an environment was created or destroyed. If the app exists the request lands; if it doesn&rsquo;t, Fly answers with its own error.</p>
<p>Three things fall out of this.</p>
<p>First, &ldquo;who&rsquo;s on staging&rdquo; stops being a question. There is <code>feature-payments-retr-3f9a1c-dev.example.com</code> and <code>fix-login-redirect-8b21e0-dev.example.com</code>, and each belongs to whoever is on that branch.</p>
<p>Second, staging becomes an honest rehearsal of prod rather than a dumping ground. It&rsquo;s built from the exact image that will go to prod, against a database cloned from a prod-shaped snapshot minutes earlier. If it works there, the only variable left in prod is prod&rsquo;s data.</p>
<p>Third, each developer gets power without getting root on anything shared. The blast radius of a branch environment is that branch environment. A fourth hire on their first day can deploy whatever they like to their branch, and the worst outcome is that their branch is broken.</p>
<figure>
  <div class="browser-frame">
    <div class="browser-frame__bar" aria-hidden="true">
      <span class="browser-frame__dots"><i></i><i></i><i></i></span>
      <span class="browser-frame__url">fly.io/dashboard/my-org</span>
    </div><span class="browser-frame__shot" ><img
      class="browser-frame__page"
      src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-apps_hu_5259e860ae14c489.webp"
      srcset="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-apps_hu_5259e860ae14c489.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-apps_hu_e5b19353521f0ce4.webp 1280w"
      sizes="(min-width: 768px) 50vw, 65vw"
      width="1400" height="900"
      alt="The Fly.io dashboard for the org: three feature-branch apps, two already suspended, next to the production apps and the shared preview Postgres and Redis"
      loading="lazy" decoding="async"
      data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-apps_hu_a766ad7bf75a9514.webp"></span></div>
  <figcaption>The org&rsquo;s app list with three branches up. Two of the three are already suspended, a minute after deploying; the shared preview Postgres and Redis sit alongside. The production apps are the rows underneath.</figcaption>
</figure>


<h2 class="relative group">Shared physically, separated logically
    <div id="shared-physically-separated-logically" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#shared-physically-separated-logically" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The tempting version of &ldquo;an environment per branch&rdquo; is a full copy: a Postgres cluster each, a Redis each, a bucket each. It&rsquo;s clean and it&rsquo;s the wrong trade. A Fly Postgres cluster is a machine plus a volume plus a restore; a Redis is another machine; you&rsquo;d be waiting minutes for each environment and paying for a dozen idle databases. The sweeper would have four kinds of thing to clean up instead of one.</p>
<p>Instead, each shared service is provisioned once for the whole preview org, sized like a small prod, and every environment gets a slice of it keyed by <code>$ENV</code>. This is the table I&rsquo;d want on the wall:</p>
<table>
	<thead>
			<tr>
					<th>Resource</th>
					<th>Shared thing</th>
					<th>What&rsquo;s per-environment</th>
					<th>Enforced by</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Compute</td>
					<td>Fly org <code>myapp-preview</code></td>
					<td>One Fly app, <code>myapp-$ENV</code>, one machine</td>
					<td>Fly: apps are the isolation unit</td>
			</tr>
			<tr>
					<td>Secrets</td>
					<td>Fly&rsquo;s per-app secret store</td>
					<td>The app&rsquo;s own secrets; <code>SECRET_KEY</code>, <code>ENCRYPTION_KEY</code> and <code>ENV_TOKEN</code> are <em>generated</em> per environment, not copied</td>
					<td>Fly: secrets belong to an app, and a machine can only read its own</td>
			</tr>
			<tr>
					<td>Postgres</td>
					<td>One Fly Postgres cluster <code>myapp-preview-pg</code></td>
					<td>One database, <code>env_feature_payments_retr_3f9a1c</code>, cloned from <code>seed_template</code></td>
					<td>Postgres: a database is a hard boundary, no cross-database queries</td>
			</tr>
			<tr>
					<td>Redis</td>
					<td>One shared Redis</td>
					<td>Key prefix <code>feature-payments-retr-3f9a1c:</code> on the cache and the Channels layer</td>
					<td>Django&rsquo;s cache <code>KEY_PREFIX</code>, the channel layer&rsquo;s <code>prefix</code></td>
			</tr>
			<tr>
					<td>Files</td>
					<td>One S3-compatible bucket <code>myapp-preview</code></td>
					<td>Object prefix <code>feature-payments-retr-3f9a1c/</code></td>
					<td>The storage backend&rsquo;s <code>location</code>; presigned URLs already scope to the key</td>
			</tr>
			<tr>
					<td>Hostname</td>
					<td>The zone&rsquo;s existing proxied <code>*.example.com</code> record</td>
					<td>The left label, <code>&lt;env&gt;-dev</code></td>
					<td>A Cloudflare Worker route on <code>*-dev.example.com/*</code></td>
			</tr>
			<tr>
					<td>Auth</td>
					<td>One dev project at the auth provider</td>
					<td>Nothing; user identity is shared on purpose (see below)</td>
					<td>Authorised domain <code>example.com</code></td>
			</tr>
			<tr>
					<td>Errors and logs</td>
					<td>The error tracker, Fly&rsquo;s logs</td>
					<td><code>environment=$ENV</code> tag on everything</td>
					<td>Settings</td>
			</tr>
	</tbody>
</table>
<p>Two of those need a word.</p>
<p><strong>Postgres: database, not schema.</strong> The other way to slice one cluster is a schema per environment, with <code>search_path</code> set per connection. Django will do it if you push a <code>-c search_path=...</code> through the connection <code>options</code>, but a connection pooler in transaction mode discards startup options unless you tell it not to, and then every connection in the pool has to belong to one environment anyway. It&rsquo;s a source of &ldquo;why is my query hitting the wrong tables&rdquo; bugs that a database per environment simply doesn&rsquo;t have. A database is a hard boundary, <code>CREATE DATABASE ... TEMPLATE</code> makes one in seconds, and <code>DROP DATABASE</code> makes it go away with nothing left behind. Branch environments connect to Postgres directly on port 5433 rather than through the pooler, the same way the background worker does in prod; the pooler earns its keep at production concurrency, which a branch environment never sees.</p>
<p><strong>Secrets: generated, not shared.</strong> It&rsquo;d be easy to have one set of dev secrets and copy it into every environment. Don&rsquo;t. The <code>env up</code> script generates a fresh <code>SECRET_KEY</code> (session signing), <code>ENCRYPTION_KEY</code> (anything encrypted at rest) and <code>ENV_TOKEN</code> (the bearer token the mobile app presents, more on that below) for each environment and stores them only in that Fly app&rsquo;s secret store. It means a session cookie from one branch is worthless on another, a leaked preview token opens one preview, and the developer never sees the values unless they go looking. If you keep secrets in a separate secrets manager rather than only in Fly, the same rule applies: the path is <code>myapp/preview/$ENV/</code>, and the sweeper deletes the path with the environment. The only secrets that <em>are</em> shared across environments are the credentials for the shared infrastructure itself (the Postgres admin URL, the Redis URL, the bucket key), and those live in CI, not in any environment.</p>
<p>The Redis row is the easiest to doubt, so here is the test. Set a cache key from inside one environment, read it from the other, then list every key in the shared Redis:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ fly ssh console -a myapp-feature-profile-tagl-fa8832 -C &#34;python manage.py shell -c \&#34;from django.core.cache import cache; cache.set(&#39;who-am-i&#39;, &#39;tagline-branch&#39;, 600)\&#34;&#34;
</span></span><span class="line"><span class="cl">$ fly ssh console -a myapp-feature-profile-webs-dcb2e7 -C &#34;python manage.py shell -c \&#34;from django.core.cache import cache; print(cache.get(&#39;who-am-i&#39;))\&#34;&#34;
</span></span><span class="line"><span class="cl">None
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ # every key in the shared Redis
</span></span><span class="line"><span class="cl">  feature-profile-tagl-fa8832:1:who-am-i
</span></span><span class="line"><span class="cl">  ...</span></span></code></pre></div></div>
<p>Same Redis, same key name, the other environment sees nothing.</p>
<p>Everything else in this post is the script that stamps <code>$ENV</code> onto those eight rows, and the rules that keep it honest.</p>

<h2 class="relative group"><code>./scripts/env up</code>
    <div id="scriptsenv-up" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#scriptsenv-up" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is the whole idea, as a script a developer runs from their branch. CI runs the same script; there&rsquo;s no separate CI path to drift. It needs <code>fly</code> and <code>git</code> on the laptop and nothing else: SQL goes to the Postgres app over <code>fly ssh</code>, and the environment&rsquo;s Redis keys and bucket objects are removed by the environment&rsquo;s own machine before it&rsquo;s destroyed, so the laptop never holds the shared Redis or bucket credentials beyond a gitignored env file. Trimmed to the lines that matter:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/usr/bin/env bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># scripts/env up|down|name|url: one environment per branch on shared preview infrastructure</span>
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail
</span></span><span class="line"><span class="cl"><span class="nv">ORG</span><span class="o">=</span><span class="si">${</span><span class="nv">PREVIEW_ORG</span><span class="k">:-</span><span class="nv">myapp</span><span class="p">-preview</span><span class="si">}</span><span class="p">;</span> <span class="nv">PG_APP</span><span class="o">=</span><span class="si">${</span><span class="nv">PREVIEW_PG_APP</span><span class="k">:-</span><span class="nv">myapp</span><span class="p">-preview-pg</span><span class="si">}</span><span class="p">;</span> <span class="nv">DOMAIN</span><span class="o">=</span><span class="si">${</span><span class="nv">BASE_DOMAIN</span><span class="k">:-</span><span class="nv">example</span><span class="p">.com</span><span class="si">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">branch</span><span class="o">=</span><span class="si">${</span><span class="nv">BRANCH</span><span class="k">:-$(</span>git rev-parse --abbrev-ref HEAD<span class="k">)</span><span class="si">}</span>
</span></span><span class="line"><span class="cl"><span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> <span class="o">==</span> <span class="s2">&#34;main&#34;</span> <span class="o">]]</span> <span class="o">&amp;&amp;</span> <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;main has staging, not a branch environment&#34;</span> &gt;<span class="p">&amp;</span>2<span class="p">;</span> <span class="nb">exit</span> 1<span class="p">;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="nv">slug</span><span class="o">=</span><span class="k">$(</span><span class="nb">printf</span> <span class="s1">&#39;%s&#39;</span> <span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> <span class="p">|</span> tr <span class="s1">&#39;[:upper:]/_ &#39;</span> <span class="s1">&#39;[:lower:]---&#39;</span> <span class="p">|</span> tr -cd <span class="s1">&#39;a-z0-9-&#39;</span> <span class="p">|</span> cut -c1-20<span class="k">)</span><span class="p">;</span> <span class="nv">slug</span><span class="o">=</span><span class="si">${</span><span class="nv">slug</span><span class="p">%-</span><span class="si">}</span>
</span></span><span class="line"><span class="cl"><span class="nv">ENV</span><span class="o">=</span><span class="si">${</span><span class="nv">ENV_OVERRIDE</span><span class="k">:-</span><span class="si">${</span><span class="nv">slug</span><span class="si">}</span><span class="p">-</span><span class="k">$(</span><span class="nb">printf</span> <span class="s1">&#39;%s&#39;</span> <span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> <span class="p">|</span> shasum -a <span class="m">256</span> <span class="p">|</span> cut -c1-6<span class="k">)</span><span class="si">}</span>   <span class="c1"># the sweeper passes the name</span>
</span></span><span class="line"><span class="cl"><span class="nv">APP</span><span class="o">=</span><span class="s2">&#34;myapp-</span><span class="si">${</span><span class="nv">ENV</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">;</span> <span class="nv">DB</span><span class="o">=</span><span class="s2">&#34;env_</span><span class="si">${</span><span class="nv">ENV</span><span class="p">//-/_</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">;</span> <span class="nv">HOST</span><span class="o">=</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">ENV</span><span class="si">}</span><span class="s2">-dev.</span><span class="si">${</span><span class="nv">DOMAIN</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">pg<span class="o">()</span> <span class="o">{</span> fly ssh console -a <span class="s2">&#34;</span><span class="nv">$PG_APP</span><span class="s2">&#34;</span> -q -C <span class="s2">&#34;psql postgres://postgres:</span><span class="si">${</span><span class="nv">PG_PASSWORD</span><span class="si">}</span><span class="s2">@localhost:5433/postgres -At -c \&#34;</span><span class="nv">$1</span><span class="s2">\&#34;&#34;</span><span class="p">;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">exists<span class="o">()</span> <span class="o">{</span> fly apps list --org <span class="s2">&#34;</span><span class="nv">$ORG</span><span class="s2">&#34;</span> --json <span class="p">|</span> grep -q <span class="s2">&#34;\&#34;Name\&#34;: *\&#34;</span><span class="nv">$APP</span><span class="s2">\&#34;&#34;</span><span class="p">;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">up<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;== </span><span class="nv">$branch</span><span class="s2"> -&gt; </span><span class="nv">$ENV</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">  exists <span class="o">||</span> fly apps create <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> --org <span class="s2">&#34;</span><span class="nv">$ORG</span><span class="s2">&#34;</span>                         <span class="c1"># idempotent</span>
</span></span><span class="line"><span class="cl">  <span class="o">[[</span> <span class="s2">&#34;</span><span class="k">$(</span>pg <span class="s2">&#34;SELECT 1 FROM pg_database WHERE datname=&#39;</span><span class="nv">$DB</span><span class="s2">&#39;&#34;</span><span class="k">)</span><span class="s2">&#34;</span> <span class="o">==</span> <span class="s2">&#34;1&#34;</span> <span class="o">]]</span> <span class="o">||</span> pg <span class="s2">&#34;CREATE DATABASE </span><span class="nv">$DB</span><span class="s2"> TEMPLATE seed_template&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nv">have</span><span class="o">=</span><span class="k">$(</span>fly secrets list -a <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> --json <span class="p">|</span> grep -o <span class="s1">&#39;&#34;Name&#34;: *&#34;[^&#34;]*&#34;&#39;</span> <span class="p">|</span> cut -d<span class="s1">&#39;&#34;&#39;</span> -f4 <span class="o">||</span> <span class="nb">true</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">  gen<span class="o">()</span> <span class="o">{</span> grep -qx <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span> <span class="o">&lt;&lt;&lt;</span><span class="s2">&#34;</span><span class="nv">$have</span><span class="s2">&#34;</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">=</span><span class="k">$(</span>openssl rand -hex 32<span class="k">)</span><span class="s2">&#34;</span><span class="p">;</span> <span class="o">}</span>   <span class="c1"># generated once, never rotated by a later `up`</span>
</span></span><span class="line"><span class="cl">  <span class="o">{</span> gen SECRET_KEY<span class="p">;</span> gen ENCRYPTION_KEY<span class="p">;</span> gen ENV_TOKEN
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;ENV_NAME=</span><span class="nv">$ENV</span><span class="s2">&#34;</span><span class="p">;</span> <span class="nb">echo</span> <span class="s2">&#34;ALLOWED_HOSTS=</span><span class="nv">$HOST</span><span class="s2">,</span><span class="nv">$APP</span><span class="s2">.fly.dev&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;DATABASE_URL=postgres://postgres:</span><span class="si">${</span><span class="nv">PG_PASSWORD</span><span class="si">}</span><span class="s2">@</span><span class="si">${</span><span class="nv">PG_APP</span><span class="si">}</span><span class="s2">.flycast:5432/</span><span class="si">${</span><span class="nv">DB</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># ... plus the shared Redis URL and bucket credentials</span>
</span></span><span class="line"><span class="cl">  <span class="o">}</span> <span class="p">|</span> fly secrets import -a <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> --stage
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  fly deploy -a <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> --config fly.preview.toml --remote-only --ha<span class="o">=</span><span class="nb">false</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --build-arg <span class="nv">GIT_SHA</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span>git rev-parse --short HEAD<span class="k">)</span><span class="s2">&#34;</span> --build-arg <span class="nv">GIT_BRANCH</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;https://</span><span class="nv">$HOST</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">down<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> <span class="o">=</span>~ ^myapp-<span class="o">[</span>a-z0-9-<span class="o">]</span>+-<span class="o">[</span>0-9a-f<span class="o">]{</span>6<span class="o">}</span>$ <span class="o">]]</span> <span class="o">||</span> <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;refusing to destroy </span><span class="nv">$APP</span><span class="s2">&#34;</span> &gt;<span class="p">&amp;</span>2<span class="p">;</span> <span class="nb">exit</span> 1<span class="p">;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> exists<span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    fly machine start -a <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> &gt;/dev/null 2&gt;<span class="p">&amp;</span><span class="m">1</span> <span class="o">||</span> <span class="nb">true</span>                 <span class="c1"># it may be suspended</span>
</span></span><span class="line"><span class="cl">    fly ssh console -a <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> -q -C <span class="s2">&#34;python manage.py teardown_env&#34;</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">&#34;warning: cleanup failed; the sweeper will catch it&#34;</span> &gt;<span class="p">&amp;</span><span class="m">2</span>
</span></span><span class="line"><span class="cl">    fly apps destroy <span class="s2">&#34;</span><span class="nv">$APP</span><span class="s2">&#34;</span> --yes
</span></span><span class="line"><span class="cl">  <span class="k">fi</span>
</span></span><span class="line"><span class="cl">  pg <span class="s2">&#34;DROP DATABASE IF EXISTS </span><span class="nv">$DB</span><span class="s2"> WITH (FORCE)&#34;</span> <span class="o">||</span> <span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="s2">&#34;</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="si">}</span><span class="s2">&#34;</span> in up<span class="o">)</span> up <span class="p">;;</span> down<span class="o">)</span> down <span class="p">;;</span> name<span class="o">)</span> <span class="nb">echo</span> <span class="s2">&#34;</span><span class="nv">$ENV</span><span class="s2">&#34;</span> <span class="p">;;</span> url<span class="o">)</span> <span class="nb">echo</span> <span class="s2">&#34;https://</span><span class="nv">$HOST</span><span class="s2">&#34;</span> <span class="p">;;</span> <span class="k">esac</span></span></span></code></pre></div></div>
<p>For the demo I ran it from my laptop rather than pushing, because the three branches were throwaways I didn&rsquo;t want in the repository&rsquo;s history and I wanted to watch the output as it went. In the team setup nobody types this: the label workflow further down runs exactly this script, and what follows is what would land in the Actions log. Here is what it printed the first time, on the branch that carries the tooling itself:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ./scripts/env up
</span></span><span class="line"><span class="cl">== feature/branch-environments -&gt; feature-branch-envir-c65cb4
</span></span><span class="line"><span class="cl">New app created: myapp-feature-branch-envir-c65cb4
</span></span><span class="line"><span class="cl">CREATE DATABASE
</span></span><span class="line"><span class="cl">Secrets have been staged, but not set on VMs. Deploy or update machines in this app for the secrets to take effect.
</span></span><span class="line"><span class="cl">==&gt; Building image
</span></span><span class="line"><span class="cl">image size: 139 MB
</span></span><span class="line"><span class="cl">✔ Machine 1a2b3c4d5e6f78 [app] update finished: success
</span></span><span class="line"><span class="cl">https://feature-branch-envir-c65cb4-dev.example.com
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ curl -s https://feature-branch-envir-c65cb4-dev.example.com/api/version/
</span></span><span class="line"><span class="cl">{&#34;environment&#34;: &#34;feature-branch-envir-c65cb4&#34;, &#34;branch&#34;: &#34;feature/branch-environments&#34;, &#34;commit&#34;: &#34;a1b2c3d4&#34;, &#34;region&#34;: &#34;syd&#34;}</span></span></code></pre></div></div>
<p>About two and a half minutes from command to URL, most of it the image build. The second <code>up</code> on the same branch, after another commit, printed no &ldquo;New app created&rdquo; and no &ldquo;CREATE DATABASE&rdquo;, staged the same secret names without regenerating the generated ones, and rebuilt: same URL, new commit.</p>
<p>Some of the choices in there deserve a sentence each.</p>
<p><strong><code>fly.preview.toml</code> is not <code>fly.toml</code>.</strong> Prod runs the API, the WebSocket server and the background worker as separate Fly apps and process groups so each scales on its own metric, which is the whole argument of the stack post. A branch environment doesn&rsquo;t need to scale; it needs to be one machine. So the preview config has a single process: the container&rsquo;s entrypoint gets a preview mode that runs migrations and then starts <em>one</em> ASGI server with WebSockets enabled. Channels already routes both HTTP and WebSocket traffic within one ASGI application, so one process serves the whole app. Sync views run thread-sensitive in that mode, roughly one request at a time, which is exactly the trade I moved away from in prod and exactly right here, because nobody testing a branch will notice.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="c"># fly.preview.toml (the app name comes from `fly deploy -a`)</span>
</span></span><span class="line"><span class="cl"><span class="nx">app</span> <span class="p">=</span> <span class="s1">&#39;placeholder&#39;</span>
</span></span><span class="line"><span class="cl"><span class="nx">primary_region</span> <span class="p">=</span> <span class="s1">&#39;syd&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">processes</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">app</span> <span class="p">=</span> <span class="s2">&#34;/app/entrypoint.sh preview&#34;</span>   <span class="c"># migrate, then one ASGI server for HTTP and WebSockets</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">env</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">DJANGO_SETTINGS_MODULE</span> <span class="p">=</span> <span class="s1">&#39;myapp.settings&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">DEBUG</span> <span class="p">=</span> <span class="s1">&#39;False&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">internal_port</span> <span class="p">=</span> <span class="mi">8000</span>
</span></span><span class="line"><span class="cl">  <span class="nx">force_https</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;suspend&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">  <span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">  <span class="p">[[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">checks</span><span class="p">]]</span>
</span></span><span class="line"><span class="cl">    <span class="nx">path</span> <span class="p">=</span> <span class="s1">&#39;/healthz&#39;</span>
</span></span><span class="line"><span class="cl">    <span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">checks</span><span class="p">.</span><span class="nx">headers</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">      <span class="nx">Host</span> <span class="p">=</span> <span class="s1">&#39;127.0.0.1&#39;</span>               <span class="c"># the app&#39;s own hostname isn&#39;t known to a static file</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">[[</span><span class="nx">vm</span><span class="p">]]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">size</span> <span class="p">=</span> <span class="s1">&#39;shared-cpu-1x&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">memory</span> <span class="p">=</span> <span class="s1">&#39;1gb&#39;</span></span></span></code></pre></div></div>
<p><strong><code>auto_stop_machines = &quot;suspend&quot;</code>, <code>min_machines_running = 0</code>.</strong> A branch environment nobody has opened since lunch should cost nothing. Suspend rather than stop gives the ~3 second resume I ended up with <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >after chasing 504s</a> instead of a 30-second cold start, and here even 30 seconds would be tolerable. Two of the three demo environments were already <code>suspended</code> in <code>fly apps list</code> a minute after their deploy finished.</p>
<figure>
  <div class="browser-frame">
    <div class="browser-frame__bar" aria-hidden="true">
      <span class="browser-frame__dots"><i></i><i></i><i></i></span>
      <span class="browser-frame__url">fly.io/apps/myapp-feature-profile-tagl-fa8832</span>
    </div><span class="browser-frame__shot" ><img
      class="browser-frame__page"
      src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-branch-app_hu_d26550a0a30599da.webp"
      srcset="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-branch-app_hu_d26550a0a30599da.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-branch-app_hu_d2f15e465075b4eb.webp 1280w"
      sizes="(min-width: 768px) 50vw, 65vw"
      width="1400" height="900"
      alt="Fly.io overview of one branch app: a single shared-cpu-1x machine in SYD, one process group, its own *.fly.dev hostname"
      loading="lazy" decoding="async"
      data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-fly-branch-app_hu_381dc63e1f9d24c0.webp"></span></div>
  <figcaption>One branch environment as Fly sees it: one machine, one process group, one region. Nothing about it is worth scaling.</figcaption>
</figure>

<p><strong><code>DEBUG = False</code>, same settings module as prod.</strong> A branch environment is a public URL. It should behave like prod in every way that matters for testing (real HTTPS, real cookies, the same health check) and unlike prod in every way that could hurt someone. Rather than a separate preview settings file, the difference is carried by the secrets <code>env up</code> sets: sandbox payment keys, no admin credentials for the auth provider, and the <code>ENV_NAME</code> that switches on every prefix below.</p>
<p><strong>One setting slices everything.</strong> <code>ENV_NAME</code> is the only new knob in the settings, and it&rsquo;s applied in four places:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">ENV_NAME</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;ENV_NAME&#34;</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">_prefix</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">ENV_NAME</span><span class="si">}</span><span class="s2">/&#34;</span> <span class="k">if</span> <span class="n">ENV_NAME</span> <span class="k">else</span> <span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">CACHES</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">][</span><span class="s2">&#34;KEY_PREFIX&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">ENV_NAME</span>                           <span class="c1"># every cache key</span>
</span></span><span class="line"><span class="cl"><span class="n">CHANNEL_LAYERS</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">][</span><span class="s2">&#34;CONFIG&#34;</span><span class="p">][</span><span class="s2">&#34;prefix&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">ENV_NAME</span><span class="si">}</span><span class="s2">:asgi&#34;</span>  <span class="c1"># Channels group and channel names</span>
</span></span><span class="line"><span class="cl"><span class="n">STORAGES</span><span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">][</span><span class="s2">&#34;OPTIONS&#34;</span><span class="p">][</span><span class="s2">&#34;location&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">_prefix</span><span class="si">}</span><span class="s2">media&#34;</span>      <span class="c1"># uploads</span>
</span></span><span class="line"><span class="cl"><span class="n">STORAGES</span><span class="p">[</span><span class="s2">&#34;staticfiles&#34;</span><span class="p">][</span><span class="s2">&#34;OPTIONS&#34;</span><span class="p">][</span><span class="s2">&#34;location&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">_prefix</span><span class="si">}</span><span class="s2">static&#34;</span></span></span></code></pre></div></div>
<p>Miss the channels one and two branches&rsquo; WebSocket servers share group names, and a message sent in one environment turns up in another. That is the kind of bug that costs a day.</p>
<p>There was a fifth place I only found by reading the shared Redis after the fact. The startup lock from the 504s post, the thing that lets one machine per image run migrations while the others wait, was keyed on the image reference. Two branch environments built from the same commit have the same image. The second one would have found the first&rsquo;s &ldquo;done&rdquo; marker and skipped its own migrations against its own, un-migrated database. Now the environment name is part of the key and the key lives under the prefix. It&rsquo;s the sort of thing a shared-infrastructure design keeps producing: anything that was implicitly &ldquo;per deployment&rdquo; has to be made explicitly per environment, and you don&rsquo;t know what was implicit until you look.</p>
<p><strong>The database is cloned, not migrated from empty.</strong> That&rsquo;s the next section.</p>

<h2 class="relative group">The database: a template, cloned in seconds
    <div id="the-database-a-template-cloned-in-seconds" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-database-a-template-cloned-in-seconds" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A branch environment&rsquo;s database can be one of three things. Shared with everyone, which is what got us here. Empty, migrated from zero and seeded with fixtures: deterministic, but fixtures are never as good as real-shaped data and are always eighteen months stale. Or a clone of a prod-shaped snapshot: realistic row counts, realistic nulls in columns the fixtures assumed were always filled, the migration that&rsquo;s instant on an empty table and takes forty minutes on a real one shows itself.</p>
<p>The third, with the second kept as a CI-only check. And what makes the third fast enough to do per branch is a Postgres feature most people forget exists:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="c1">-- Nightly, on the preview cluster. Restore last night&#39;s prod backup into seed_raw,
</span></span></span><span class="line"><span class="cl"><span class="c1">-- run the anonymiser, then freeze it as a template nobody can connect to.
</span></span></span><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">seed_raw</span><span class="w"> </span><span class="k">RENAME</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="n">seed_template</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">UPDATE</span><span class="w"> </span><span class="n">pg_database</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">datistemplate</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">true</span><span class="p">,</span><span class="w"> </span><span class="n">datallowconn</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">false</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">datname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;seed_template&#39;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">-- Per branch, in the time it takes to copy the files:
</span></span></span><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">env_feature_payments_retr_3f9a1c</span><span class="w"> </span><span class="k">TEMPLATE</span><span class="w"> </span><span class="n">seed_template</span><span class="p">;</span></span></span></code></pre></div></div>
<p><code>CREATE DATABASE ... TEMPLATE</code> is a file-level copy within the cluster. A few gigabytes is seconds, not the minutes a <code>pg_restore</code> takes. The template can&rsquo;t have open connections while it&rsquo;s being cloned, which is what <code>datallowconn = false</code> enforces, and it&rsquo;s also why the nightly job builds into <code>seed_raw</code> and renames at the end rather than restoring over the live template.</p>
<p>For the demo the template was built by hand rather than from a prod dump: a tunnel to the preview cluster with <code>fly proxy</code>, <code>manage.py migrate</code>, then the repo&rsquo;s own seed commands for a small but realistic data set. Nineteen megabytes. Three clones of it appeared in the cluster as three branches came up, each 19 MB, each taking under a second:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ fly ssh console -a myapp-preview-pg -C &#34;psql ... -c &#39;\l&#39;&#34;
</span></span><span class="line"><span class="cl">             datname             | template | size
</span></span><span class="line"><span class="cl">---------------------------------+----------+-------
</span></span><span class="line"><span class="cl"> env_feature_branch_envir_c65cb4 | f        | 19 MB
</span></span><span class="line"><span class="cl"> env_feature_profile_tagl_fa8832 | f        | 19 MB
</span></span><span class="line"><span class="cl"> env_feature_profile_webs_dcb2e7 | f        | 19 MB
</span></span><span class="line"><span class="cl"> seed_template                   | t        | 19 MB</span></span></code></pre></div></div>
<p>The preview cluster is one unmanaged Fly Postgres node, <code>shared-cpu-1x</code> with a 1 GB volume, about two dollars a month. The branch environments connect to it over <code>.flycast</code> on the private network; nothing about it is public.</p>
<p>The anonymiser is not optional. The moment prod data is reachable on a <code>-dev</code> hostname, it needs to not be prod data: names, emails, phone numbers, addresses, free text, payment references, tokens, all overwritten before the rename. If the anonymiser fails, the previous night&rsquo;s template stays and someone gets a message. It&rsquo;s a small amount of work and it&rsquo;s the thing that makes the whole approach acceptable to whoever answers your privacy questionnaire.</p>
<p>The entrypoint runs <code>migrate</code> on the first boot of every image, so the branch&rsquo;s own migrations apply on top of the template. That&rsquo;s the first honest test of &ldquo;does my migration work on data shaped like prod&rdquo;, and it happens before anyone looks at the URL.</p>

<h2 class="relative group">Hostnames: a wildcard, a Worker, and two things the docs didn&rsquo;t tell me
    <div id="hostnames-a-wildcard-a-worker-and-two-things-the-docs-didnt-tell-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#hostnames-a-wildcard-a-worker-and-two-things-the-docs-didnt-tell-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Every environment needs a public HTTPS hostname, and &ldquo;public&rdquo; is the point: a tester on a phone, a designer on another network, a product manager who won&rsquo;t install a VPN. Fly gives each app <code>myapp-$ENV.fly.dev</code> for free, and that alone works. But <code>*.fly.dev</code> hostnames have three problems: they&rsquo;re ugly to send around, Cloudflare isn&rsquo;t in front of them so none of the protections you have on prod apply, and if you want them on your own domain you&rsquo;re doing a DNS record and a certificate per environment, which is one more thing for the sweeper.</p>
<p>The version that removes all of that is one wildcard and one Cloudflare Worker. I&rsquo;d planned <code>*.dev.example.com</code>. Two things stopped that, and both are worth knowing before you plan the same.</p>
<p><strong>Universal SSL covers one wildcard level.</strong> Cloudflare&rsquo;s free certificate is issued for <code>example.com</code> and <code>*.example.com</code>. It does not cover <code>*.dev.example.com</code>; a second-level wildcard needs Advanced Certificate Manager at $10 a month. The first request to <code>feature-branch-envir-c65cb4.dev.example.com</code> died with an SSL handshake failure before anything of mine ran. The fix is to keep the environment in the first label: <code>feature-branch-envir-c65cb4-dev.example.com</code>. That&rsquo;s covered by the certificate that already exists, and by the proxied <code>*.example.com</code> record that already exists, the one I set up as a sinkhole for <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/#the-spike-was-a-red-herring" >the scanners walking invented subdomains</a>. So an environment needs no DNS at all.</p>
<p><strong>Redirect Rules run before Workers.</strong> With the route in place, every <code>-dev</code> hostname answered 301 to the www homepage without the Worker running. The sinkhole is a zone-level redirect rule, &ldquo;any other subdomain → www&rdquo;, and Cloudflare&rsquo;s redirect phase executes before the Workers phase. The rule needed one clause: <code>and not ends_with(http.host, &quot;-dev.example.com&quot;)</code>. The deploy script adds it on <code>up</code> and removes it on <code>down</code>, because a rule someone edited by hand is the thing that gets forgotten.</p>
<figure>
  <div class="browser-frame">
    <div class="browser-frame__bar" aria-hidden="true">
      <span class="browser-frame__dots"><i></i><i></i><i></i></span>
      <span class="browser-frame__url">dash.cloudflare.com/…/example.com/rules/redirect-rules/…/edit</span>
    </div><span class="browser-frame__shot" ><img
      class="browser-frame__page"
      src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-redirect-rules_hu_4f6c3fdde94f28d6.webp"
      srcset="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-redirect-rules_hu_4f6c3fdde94f28d6.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-redirect-rules_hu_38e5e373828b8373.webp 1280w"
      sizes="(min-width: 768px) 50vw, 65vw"
      width="1400" height="900"
      alt="The redirect rule in the Cloudflare dashboard, with the expression ending in the -dev exception"
      loading="lazy" decoding="async"
      data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-redirect-rules_hu_71d546da5b5668b6.webp"></span></div>
  <figcaption>The sinkhole redirect after the script has been at it. Everything under the zone still bounces to www except hostnames ending in <code>-dev.example.com</code>, which fall through to the Worker.</figcaption>
</figure>

<p>The DNS side is the record that was already there:</p>
<figure>
  <div class="browser-frame">
    <div class="browser-frame__bar" aria-hidden="true">
      <span class="browser-frame__dots"><i></i><i></i><i></i></span>
      <span class="browser-frame__url">dash.cloudflare.com/…/example.com/dns/records</span>
    </div><span class="browser-frame__shot" ><img
      class="browser-frame__page"
      src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-dns_hu_6d425bb8f7651010.webp"
      srcset="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-dns_hu_6d425bb8f7651010.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-dns_hu_2c615b9e52444abd.webp 1280w"
      sizes="(min-width: 768px) 50vw, 65vw"
      width="1400" height="900"
      alt="The zone&#39;s DNS records filtered to the wildcard: one proxied A record for the zone&#39;s wildcard pointing at 192.0.2.1"
      loading="lazy" decoding="async"
      data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-dns_hu_952422604a6c8304.webp"></span></div>
  <figcaption>The only DNS record involved, and it predates this post. A proxied wildcard at a placeholder address: Cloudflare answers, and what happens next is up to rules and Workers.</figcaption>
</figure>

<p>With those two out of the way the routing is a single Worker on <code>*-dev.example.com/*</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="c1">// dev-router.js: &lt;env&gt;-dev.example.com -&gt; myapp-&lt;env&gt;.fly.dev
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">DEV_SUFFIX</span> <span class="o">=</span> <span class="s2">&#34;-dev.example.com&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ENV_NAME</span> <span class="o">=</span> <span class="sr">/^[a-z0-9-]+-[0-9a-f]{6}$/</span><span class="p">;</span>   <span class="c1">// slug-hash, as scripts/env names them
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="k">default</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">async</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">request</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">url</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">URL</span><span class="p">(</span><span class="nx">request</span><span class="p">.</span><span class="nx">url</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">url</span><span class="p">.</span><span class="nx">hostname</span><span class="p">.</span><span class="nx">endsWith</span><span class="p">(</span><span class="nx">DEV_SUFFIX</span><span class="p">))</span> <span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="s2">&#34;not a dev hostname&#34;</span><span class="p">,</span> <span class="p">{</span> <span class="nx">status</span><span class="o">:</span> <span class="mi">404</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">env</span> <span class="o">=</span> <span class="nx">url</span><span class="p">.</span><span class="nx">hostname</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="nx">DEV_SUFFIX</span><span class="p">.</span><span class="nx">length</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">ENV_NAME</span><span class="p">.</span><span class="nx">test</span><span class="p">(</span><span class="nx">env</span><span class="p">))</span> <span class="k">return</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="sb">`no such environment: </span><span class="si">${</span><span class="nx">env</span><span class="si">}</span><span class="sb">`</span><span class="p">,</span> <span class="p">{</span> <span class="nx">status</span><span class="o">:</span> <span class="mi">404</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="nx">url</span><span class="p">.</span><span class="nx">hostname</span> <span class="o">=</span> <span class="sb">`myapp-</span><span class="si">${</span><span class="nx">env</span><span class="si">}</span><span class="sb">.fly.dev`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">upstream</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Request</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span> <span class="nx">request</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="nx">upstream</span><span class="p">.</span><span class="nx">headers</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s2">&#34;X-Forwarded-Host&#34;</span><span class="p">,</span> <span class="nx">request</span><span class="p">.</span><span class="nx">headers</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s2">&#34;host&#34;</span><span class="p">)</span> <span class="o">??</span> <span class="s2">&#34;&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">res</span> <span class="o">=</span> <span class="kr">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">upstream</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">out</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Response</span><span class="p">(</span><span class="nx">res</span><span class="p">.</span><span class="nx">body</span><span class="p">,</span> <span class="nx">res</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="nx">out</span><span class="p">.</span><span class="nx">headers</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s2">&#34;X-Robots-Tag&#34;</span><span class="p">,</span> <span class="s2">&#34;noindex, nofollow&#34;</span><span class="p">);</span>   <span class="c1">// a half-finished branch is not for search engines
</span></span></span><span class="line"><span class="cl">    <span class="nx">out</span><span class="p">.</span><span class="nx">headers</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s2">&#34;X-Dev-Environment&#34;</span><span class="p">,</span> <span class="nx">env</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nx">out</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span></span></span></code></pre></div></div>
<p>Cloudflare terminates TLS on the wildcard, the Worker computes the Fly hostname from the label, and Fly&rsquo;s own <code>*.fly.dev</code> certificate covers the hop to the machine. WebSocket upgrades pass through <code>fetch</code> unchanged. And because the route is on the zone, everything Cloudflare does for prod is available: WAF rules, rate limits, bot fight mode, and Access.</p>
<figure>
  <div class="browser-frame">
    <div class="browser-frame__bar" aria-hidden="true">
      <span class="browser-frame__dots"><i></i><i></i><i></i></span>
      <span class="browser-frame__url">dash.cloudflare.com/…/example.com/workers</span>
    </div><span class="browser-frame__shot" ><img
      class="browser-frame__page"
      src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-zone-worker-routes_hu_bd62e960f954de34.webp"
      srcset="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-zone-worker-routes_hu_bd62e960f954de34.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-zone-worker-routes_hu_9996f094fe4379cd.webp 1280w"
      sizes="(min-width: 768px) 50vw, 65vw"
      width="1400" height="900"
      alt="Cloudflare&#39;s Workers Routes page for the zone: one route, *-dev.example.com/*, bound to the dev-router Worker"
      loading="lazy" decoding="async"
      data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/dash-cf-zone-worker-routes_hu_78420fc686a1e6e3.webp"></span></div>
  <figcaption>The entire Cloudflare side of an environment: one route pattern, one Worker. No environment is named anywhere in this dashboard.</figcaption>
</figure>

<p>Three branches up at once, one Worker, and what each hostname answered:</p>
<table>
	<thead>
			<tr>
					<th>Hostname</th>
					<th>HTTP</th>
					<th>Routed to</th>
					<th><code>branch</code></th>
					<th><code>commit</code></th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>feature-branch-envir-c65cb4-dev.example.com</code></td>
					<td>200</td>
					<td><code>myapp-feature-branch-envir-c65cb4.fly.dev</code></td>
					<td><code>feature/branch-environments</code></td>
					<td><code>a1b2c3d4</code></td>
			</tr>
			<tr>
					<td><code>feature-profile-tagl-fa8832-dev.example.com</code></td>
					<td>200</td>
					<td><code>myapp-feature-profile-tagl-fa8832.fly.dev</code></td>
					<td><code>feature/profile-tagline</code></td>
					<td><code>e5f6a7b8</code></td>
			</tr>
			<tr>
					<td><code>feature-profile-webs-dcb2e7-dev.example.com</code></td>
					<td>200</td>
					<td><code>myapp-feature-profile-webs-dcb2e7.fly.dev</code></td>
					<td><code>feature/profile-website</code></td>
					<td><code>c9d0e1f2</code></td>
			</tr>
			<tr>
					<td><code>not-an-env-dev.example.com</code></td>
					<td>404</td>
					<td>nothing; the Worker refuses the label</td>
					<td></td>
					<td></td>
			</tr>
	</tbody>
</table>
<p>Every 200 came back with <code>X-Robots-Tag: noindex, nofollow</code> and <code>X-Dev-Environment: &lt;env&gt;</code> added by the Worker. The 404 is the Worker&rsquo;s own, before Fly is ever asked.</p>
<p>Cloudflare Access is what turns &ldquo;public&rdquo; into &ldquo;reachable by the people who should reach it&rdquo;. One application on <code>*-dev.example.com</code> with a policy of &ldquo;anyone with an @example.com email&rdquo; and browser visitors get a login page once and never think about it again. That covers designers, PMs and testers on laptops. I didn&rsquo;t add it for the demo; the environments were up for an hour with <code>noindex</code> and nothing on them but seed data.</p>
<p>It doesn&rsquo;t cover the mobile app, because a native app can&rsquo;t complete an Access login redirect. Two ways through. Access supports service tokens (a <code>CF-Access-Client-Id</code> / <code>CF-Access-Client-Secret</code> header pair) that bypass the login, and the mobile app&rsquo;s dev menu can carry a pair for the dev hostnames. Or, simpler, add an Access bypass for <code>/api/</code> and have Django reject any request without either a signed-in user or the environment&rsquo;s <code>ENV_TOKEN</code>, the one <code>env up</code> generated for exactly this environment. Either way, the mobile side is a dev-menu screen with two fields, base URL and token, and the app a tester already has installed is now talking to that branch. No new build. The <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-02-22-self-hosted-ota-updates/" >over-the-air update channel</a> stays pointed at whatever the app was built with; the API base URL is a runtime setting, not a build-time one, and it should have been anyway.</p>
<p>Auth is the one shared service that&rsquo;s shared on purpose. Identity isn&rsquo;t environment-specific; a tester wants to log in with the same account on every branch. One dev project at the auth provider with <code>example.com</code> in its authorised domains covers every <code>-dev</code> hostname, and every environment verifies ID tokens against it. The users table in each environment&rsquo;s database is a copy of the template&rsquo;s users, so the tester&rsquo;s account exists everywhere the template did.</p>

<h2 class="relative group">Automating preview environments with GitHub Actions: events in, environments out
    <div id="automating-preview-environments-with-github-actions-events-in-environments-out" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#automating-preview-environments-with-github-actions-events-in-environments-out" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Everything above by hand is fine when you&rsquo;re the only one doing it. The moment there&rsquo;s a second developer it has to happen because something happened in git, not because someone remembered. The whole automation is four workflows, one repository setting, one ruleset and one GitHub environment, and every one of them maps a git event to a call of the same <code>scripts/env</code> the developer runs by hand. There is no CI-only path to drift.</p>
<table>
	<thead>
			<tr>
					<th>Git event</th>
					<th>What runs</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>PR gets the <code>env</code> label</td>
					<td><code>branch-env.yml</code> → <code>scripts/env up</code></td>
					<td>creates that branch&rsquo;s environment, posts the URL on the PR</td>
			</tr>
			<tr>
					<td>push to a labelled PR&rsquo;s branch</td>
					<td><code>branch-env.yml</code> → <code>scripts/env up</code></td>
					<td>updates it in place</td>
			</tr>
			<tr>
					<td>label removed, PR closed or merged</td>
					<td><code>branch-env.yml</code> → <code>scripts/env down</code></td>
					<td>the environment cleans up its slice and is destroyed</td>
			</tr>
			<tr>
					<td>branch deleted</td>
					<td><code>branch-env.yml</code> → <code>scripts/env down</code></td>
					<td>backstop for a branch brought up by hand that never had a PR; the repo setting <em>delete branch on merge</em> makes merging fire it too</td>
			</tr>
			<tr>
					<td>push to any other branch</td>
					<td>nothing</td>
					<td>a backup push or a spike costs nothing until someone asks to see it</td>
			</tr>
			<tr>
					<td>pull request to <code>main</code></td>
					<td><code>migrations-check.yml</code></td>
					<td>one leaf in the migration graph after merging <code>main</code>; full chain applies from empty</td>
			</tr>
			<tr>
					<td>tag <code>v*</code> pushed</td>
					<td><code>release.yml</code></td>
					<td>staging environment from the tag; a reviewer&rsquo;s approval promotes the same image to prod and destroys staging</td>
			</tr>
			<tr>
					<td>02:00 nightly</td>
					<td><code>sweep-envs.yml</code> → <code>scripts/sweep-envs</code></td>
					<td>destroys anything whose branch is gone or idle a week</td>
			</tr>
	</tbody>
</table>

<h3 class="relative group">The label, push and delete workflow
    <div id="the-label-push-and-delete-workflow" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-label-push-and-delete-workflow" aria-label="Anchor">#</a>
    </span>
    
</h3>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/branch-env.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">branch environment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">pull_request</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">types</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">labeled, unlabeled, synchronize, closed]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">delete</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">workflow_dispatch</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">inputs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">branch</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="nt">branch to bring up without a PR, required</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">concurrency</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">env-${{ github.head_ref || github.event.ref || inputs.branch }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cancel-in-progress</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">FLY_API_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.FLY_PREVIEW_TOKEN }}  </span><span class="w"> </span><span class="c"># org-scoped deploy token; a separate preview org keeps it away from prod</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">PG_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.PREVIEW_PG_PASSWORD }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="c"># ... the shared Redis URL and bucket credentials, the same way</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">up</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># the gate: a PR carrying the `env` label, or someone asking for a branch by hand</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="p">&gt;-</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      github.event_name == &#39;workflow_dispatch&#39; ||
</span></span></span><span class="line"><span class="cl"><span class="sd">      (github.event_name == &#39;pull_request&#39; &amp;&amp; github.event.action != &#39;closed&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">        &amp;&amp; contains(github.event.pull_request.labels.*.name, &#39;env&#39;))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">ref</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;${{ github.head_ref || inputs.branch }}&#34;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">superfly/flyctl-actions/setup-flyctl@master</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">env</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">BRANCH</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;${{ github.head_ref || inputs.branch }}&#34;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">scripts/env up &amp;&amp; echo &#34;url=$(scripts/env url)&#34; &gt;&gt; &#34;$GITHUB_OUTPUT&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">github.event_name == &#39;pull_request&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">marocchino/sticky-pull-request-comment@v2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">header</span><span class="p">:</span><span class="w"> </span><span class="l">env</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">message</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Branch environment: ${{ steps.env.outputs.url }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">down</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># label removed, labelled PR closed or merged, or the branch itself deleted</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="p">&gt;-</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      (github.event_name == &#39;pull_request&#39; &amp;&amp;
</span></span></span><span class="line"><span class="cl"><span class="sd">        ((github.event.action == &#39;closed&#39; &amp;&amp; contains(github.event.pull_request.labels.*.name, &#39;env&#39;)) ||
</span></span></span><span class="line"><span class="cl"><span class="sd">         (github.event.action == &#39;unlabeled&#39; &amp;&amp; github.event.label.name == &#39;env&#39;))) ||
</span></span></span><span class="line"><span class="cl"><span class="sd">      (github.event_name == &#39;delete&#39; &amp;&amp; github.event.ref_type == &#39;branch&#39;)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">superfly/flyctl-actions/setup-flyctl@master</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">BRANCH=&#34;${{ github.head_ref || github.event.ref }}&#34; scripts/env down</span></span></span></code></pre></div></div>
<p>The gate is the <code>env</code> label, and it&rsquo;s the part I&rsquo;d argue for hardest. Without one, every branch costs an app and a database clone: the branch someone pushes to back up a laptop, the spike that gets squashed tomorrow, the dependency bump a bot opened at 3 a.m. So a push alone does nothing and opening a PR does nothing. Adding <code>env</code> to the PR brings the environment up, every push while the label is on updates it, and removing the label, closing the PR or merging it takes it down. Opting in is still a GitHub event, not a request to a person, and it&rsquo;s one click made by the one person who knows whether anyone is going to look. The sticky comment is the user interface: label the PR, a comment appears with a URL they can send to anyone, and it updates in place on every push rather than stacking. <code>workflow_dispatch</code> covers the branch you want up before there&rsquo;s a PR, and <code>./scripts/env up</code> from a laptop covers the rest.</p>
<p>Two other gates are worth knowing. A branch-name filter (<code>branches: ['feature/**', 'fix/**']</code>) needs no PR but makes every feature branch pay. &ldquo;Any PR that isn&rsquo;t a draft&rdquo; is the most automatic, and the right choice if your team opens PRs late and only when they&rsquo;re ready. I went with the label because it puts the cost decision with the person who has the context, and because it can be taken back without closing anything. One side effect I like: <code>pull_request</code> runs from forks don&rsquo;t get secrets, so a fork can&rsquo;t bring up an environment.</p>
<p><code>concurrency</code> matters more than it looks. A run of quick pushes to one branch cancels the earlier deploys rather than racing them; two <code>fly deploy</code>s to the same app at once end with one of them losing.</p>
<p><code>closed</code> covers teardown for anything that had a PR, merged or not. The <code>delete</code> event is the backstop for a branch brought up by <code>workflow_dispatch</code> that never got one, and it only fires if branches actually get deleted. That&rsquo;s a repository setting, <em>Automatically delete head branches</em>, and it&rsquo;s off by default. With it on, clicking Merge closes the PR and deletes the branch, and either event alone would have run <code>env down</code>. The merge button is the teardown.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">gh api -X PATCH repos/org/myapp -f <span class="nv">delete_branch_on_merge</span><span class="o">=</span>true</span></span></code></pre></div></div>
<p>The <code>FLY_PREVIEW_TOKEN</code> is the dangerous line in the file. It can create and destroy apps, and if <code>$APP</code> were ever anything but a branch environment it could destroy prod. Two guards, and I&rsquo;d keep both: the token is scoped to a separate Fly org so it literally cannot see the production org (for the demo I used the production org, to skip setting up billing on a new one, and relied on the second guard alone); and <code>down()</code> refuses any app name that doesn&rsquo;t match the <code>slug-hash</code> pattern.</p>

<h3 class="relative group">The check a branch environment can&rsquo;t do for itself
    <div id="the-check-a-branch-environment-cant-do-for-itself" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-check-a-branch-environment-cant-do-for-itself" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>A branch only ever sees its own migrations, so the check that matters runs on the pull request against a merge with <code>main</code>, and it&rsquo;s the one status check <code>main</code> requires:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/migrations-check.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">migrations</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">pull_request</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">main]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">check</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">postgres</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">postgres:18</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="nt">ci, POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l">ci }</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;5432:5432&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">options</span><span class="p">:</span><span class="w"> </span>--<span class="l">health-cmd pg_isready --health-interval 5s --health-retries 10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">DATABASE_URL</span><span class="p">:</span><span class="w"> </span><span class="l">postgres://postgres:ci@localhost:5432/ci</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">SECRET_KEY</span><span class="p">:</span><span class="w"> </span><span class="l">ci-only</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">fetch-depth</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-python@v5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">python-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;3.14&#39;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">pip install -r requirements.txt</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Merge main into the branch (a conflict here fails the check too)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          git config user.email ci@example.com &amp;&amp; git config user.name ci
</span></span></span><span class="line"><span class="cl"><span class="sd">          git merge --no-edit origin/main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">One leaf in the migration graph</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">python manage.py makemigrations --check --dry-run</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">The full chain applies from nothing</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">python manage.py migrate --noinput</span></span></span></code></pre></div></div>
<p>Thirty seconds. It&rsquo;s what turns the <code>0042</code> / <code>0042</code> collision from a release-day surprise into a red check on the second developer&rsquo;s PR; the migrations section below shows it firing.</p>

<h3 class="relative group">The rules on <code>main</code>
    <div id="the-rules-on-main" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-rules-on-main" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>None of this holds up if branches live for a month or if someone pushes to <code>main</code> directly. The rules are less about protection and more about keeping environments small and their migrations current:</p>
<ul>
<li><strong><code>main</code> is protected and always deployable.</strong> Merge via PR with the <code>migrations</code> check green <em>against the current <code>main</code></em> (strict status checks, so a PR that was green last week re-runs after <code>main</code> moves). Linear history, no force-push, no deletion.</li>
<li><strong>Feature branches are short.</strong> A branch open for two weeks has an environment two weeks behind <code>main</code> and a migration two weeks more likely to collide with someone else&rsquo;s. A month-long feature merges to <code>main</code> behind a flag in pieces. The flag is the long-lived thing, not the branch.</li>
<li><strong>One branch, one environment, one database.</strong> Already enforced by the naming.</li>
<li><strong>Release tags are the only thing that goes to prod.</strong> The image is built once from the tag; staging and prod run that image, not a rebuild.</li>
<li><strong>A PR that touches <code>migrations/</code> is reviewed by one of the other two.</strong> Below.</li>
</ul>
<p>As a ruleset, applied once with <code>gh api -X POST repos/org/myapp/rulesets --input ruleset-main.json</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;main&#34;</span><span class="p">,</span> <span class="nt">&#34;target&#34;</span><span class="p">:</span> <span class="s2">&#34;branch&#34;</span><span class="p">,</span> <span class="nt">&#34;enforcement&#34;</span><span class="p">:</span> <span class="s2">&#34;active&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;conditions&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;ref_name&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;include&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;~DEFAULT_BRANCH&#34;</span><span class="p">],</span> <span class="nt">&#34;exclude&#34;</span><span class="p">:</span> <span class="p">[]</span> <span class="p">}</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;rules&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;deletion&#34;</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;non_fast_forward&#34;</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;required_linear_history&#34;</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;pull_request&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;parameters&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;required_approving_review_count&#34;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="nt">&#34;dismiss_stale_reviews_on_push&#34;</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                      <span class="nt">&#34;required_review_thread_resolution&#34;</span><span class="p">:</span> <span class="kc">true</span> <span class="p">}</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;required_status_checks&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;parameters&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;strict_required_status_checks_policy&#34;</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                      <span class="nt">&#34;required_status_checks&#34;</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="nt">&#34;context&#34;</span><span class="p">:</span> <span class="s2">&#34;check&#34;</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>One honest wrinkle. This repository is private on GitHub&rsquo;s Free plan, and on Free, branch protection and rulesets are only available for public repositories; the API answers <code>403 Upgrade to GitHub Pro or make this repository public</code>. The <code>delete_branch_on_merge</code> setting, the environments and the workflows all work on Free. So for a small team on a private Free repo, the ruleset above is a convention: the <code>migrations</code> workflow still runs and still goes red, but nothing greys out the merge button. That&rsquo;s a $4-a-month decision, and I&rsquo;d make it the moment a second person can merge.</p>
<p>There&rsquo;s no <code>develop</code> branch and no <code>release/*</code> branches. Git-flow was designed around integration being expensive and needing to be batched. Branch environments make integration cheap, so the batching goes.</p>

<h3 class="relative group">Staging is a tag&rsquo;s environment, and prod is a promotion
    <div id="staging-is-a-tags-environment-and-prod-is-a-promotion" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#staging-is-a-tags-environment-and-prod-is-a-promotion" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Staging reuses the whole machinery. A <code>v*</code> tag on <code>main</code> runs <code>scripts/env up</code> with the branch name <code>staging/v1.42.0</code>, which the naming turns into <code>staging-v1-42-0-&lt;hash&gt;</code>: same clone-from-template database, same suspend-when-idle machine, same hostname scheme. The difference is what comes after:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/release.yml (abridged)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">push</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">tags</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;v*&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">staging</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">BRANCH=&#34;staging/${GITHUB_REF_NAME}&#34; scripts/env up</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">echo &#34;image=$(fly image show -a &#34;myapp-$(BRANCH=staging/$GITHUB_REF_NAME scripts/env name)&#34; --json | jq -r &#39;…&#39;)&#34; &gt;&gt; &#34;$GITHUB_OUTPUT&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">production</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">needs</span><span class="p">:</span><span class="w"> </span><span class="l">staging</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span><span class="l">production       </span><span class="w"> </span><span class="c"># required reviewer on this environment = the promotion gate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">fly deploy --config fly.api.toml --image &#34;${{ needs.staging.outputs.image }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">fly deploy --config fly.ws.toml  --image &#34;${{ needs.staging.outputs.image }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">BRANCH=&#34;staging/${GITHUB_REF_NAME}&#34; scripts/env down</span></span></span></code></pre></div></div>
<p>The <code>production</code> job waits on a GitHub environment with a required reviewer. Someone looks at staging, clicks Approve, and prod receives the <em>same image</em> staging ran, by reference, not a rebuild. The last step destroys staging because it&rsquo;s done its job. If nobody approves, the release candidate was rejected, and the sweeper removes staging when its &ldquo;branch&rdquo; (<code>staging/v1.42.0</code> never exists on origin) fails the liveness check. Nothing gets to exist without a reason the sweeper can check.</p>

<h3 class="relative group">The nightly sweeper
    <div id="the-nightly-sweeper" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-nightly-sweeper" aria-label="Anchor">#</a>
    </span>
    
</h3>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .github/workflows/sweep-envs.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">schedule</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">cron</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0 14 * * *&#39;</span><span class="w">     </span><span class="c"># 02:00 NZST</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">workflow_dispatch</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">sweep</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v4</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">fetch-depth</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">superfly/flyctl-actions/setup-flyctl@master</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">scripts/sweep-envs</span></span></span></code></pre></div></div>
<p><code>fetch-depth: 0</code> because the sweeper needs every remote branch and its last commit date, not the one commit Actions checks out by default. <code>workflow_dispatch</code> so it can be run by hand after an incident.</p>

<h3 class="relative group">What a developer actually does
    <div id="what-a-developer-actually-does" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-a-developer-actually-does" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Push a branch, open a PR, and put the <code>env</code> label on it when they want someone to look. That&rsquo;s the list. A URL appears on the PR within three minutes, updates on every push, and disappears when the PR is merged or closed. The migration check tells them before anyone else does if they&rsquo;ve collided with a teammate. Nobody claims staging because there is no staging to claim, and nobody tears anything down because tearing down is what merging does.</p>
<p>And what the person who set it up does per branch, afterwards: nothing. The shared Postgres, Redis and bucket are provisioned once, the secrets go into Actions once, and from then on the recurring work is reading the sweeper&rsquo;s log on a Monday. No ticket queue, no bot to ask, nobody to prompt.</p>

<h2 class="relative group">Teardown that doesn&rsquo;t trust the teardown
    <div id="teardown-that-doesnt-trust-the-teardown" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#teardown-that-doesnt-trust-the-teardown" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>env down</code> is the ordinary case, and the interesting part of it is <em>who</em> does the cleaning. The Redis keys and the bucket objects belong to the shared services, and the laptop running <code>down</code> doesn&rsquo;t need credentials for either, because the environment&rsquo;s own machine already has them and already knows its prefix. So <code>down</code> starts the machine if it&rsquo;s suspended, runs a small cleanup command on it, and only then destroys the app and drops the database. The environment cleans up after itself:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ BRANCH=feature/profile-website ./scripts/env down
</span></span><span class="line"><span class="cl">redis: deleted 0 keys under feature-profile-webs-dcb2e7:*
</span></span><span class="line"><span class="cl">bucket: deleted 278 objects under feature-profile-webs-dcb2e7/
</span></span><span class="line"><span class="cl">Destroyed app myapp-feature-profile-webs-dcb2e7
</span></span><span class="line"><span class="cl">DROP DATABASE
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ curl -s -o /dev/null -w &#39;%{http_code}\n&#39; https://feature-profile-webs-dcb2e7-dev.example.com/api/version/
</span></span><span class="line"><span class="cl">530</span></span></code></pre></div></div>
<p>The 530 is the point about coordination again: the Worker still routes the hostname, Fly has nothing there, and nobody had to tell Cloudflare.</p>
<p>The cleanup command refuses to run without <code>ENV_NAME</code>, because with an empty prefix it would be a wipe of the shared bucket. That guard is the whole reason it exists as a command inside the app rather than three lines in the shell script.</p>
<p>Environments leak anyway. A branch is deleted while GitHub Actions is having an outage. A developer runs <code>env up</code> from a laptop for a spike and never pushes. A <code>fly apps destroy</code> fails transiently and the <code>|| true</code> swallows it. Three months later there are forty apps and a bill that makes someone ask what happened.</p>
<p>So there&rsquo;s a second mechanism that doesn&rsquo;t trust the first one: the nightly sweeper from the table above. It lists every <code>myapp-*</code> app, works out which branch each one belongs to, and destroys any whose branch is gone or hasn&rsquo;t been pushed to in seven days. A week-idle branch doesn&rsquo;t need a live URL; the next push recreates it in two minutes. A hash isn&rsquo;t reversible, so the sweeper computes the environment name for every live branch and destroys any app that isn&rsquo;t in that set, then drops any <code>env_*</code> database with no app behind it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/usr/bin/env bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># scripts/sweep-envs: nightly</span>
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail
</span></span><span class="line"><span class="cl"><span class="nv">ORG</span><span class="o">=</span><span class="si">${</span><span class="nv">PREVIEW_ORG</span><span class="k">:-</span><span class="nv">myapp</span><span class="p">-preview</span><span class="si">}</span><span class="p">;</span> <span class="nv">TTL_DAYS</span><span class="o">=</span><span class="si">${</span><span class="nv">TTL_DAYS</span><span class="k">:-</span><span class="nv">7</span><span class="si">}</span>
</span></span><span class="line"><span class="cl"><span class="nv">cutoff</span><span class="o">=</span><span class="k">$((</span> <span class="k">$(</span>date +%s<span class="k">)</span> <span class="o">-</span> TTL_DAYS*86400 <span class="k">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">git fetch --prune --quiet origin
</span></span><span class="line"><span class="cl"><span class="nv">live</span><span class="o">=</span><span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="nv">IFS</span><span class="o">=</span><span class="s1">$&#39;\t&#39;</span> <span class="nb">read</span> -r ts branch<span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">  <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> <span class="o">==</span> <span class="s2">&#34;main&#34;</span> <span class="o">]]</span> <span class="o">&amp;&amp;</span> <span class="k">continue</span>
</span></span><span class="line"><span class="cl">  <span class="o">((</span> ts &gt;<span class="o">=</span> cutoff <span class="o">))</span> <span class="o">||</span> <span class="k">continue</span>
</span></span><span class="line"><span class="cl">  <span class="nv">live</span><span class="o">+=</span><span class="s2">&#34;</span><span class="k">$(</span><span class="nv">BRANCH</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$branch</span><span class="s2">&#34;</span> scripts/env name<span class="k">)</span><span class="s2">&#34;</span><span class="s1">$&#39;\n&#39;</span>
</span></span><span class="line"><span class="cl"><span class="k">done</span> &lt; &lt;<span class="o">(</span>git <span class="k">for</span>-each-ref --format<span class="o">=</span><span class="s1">&#39;%(committerdate:unix)%09%(refname:short)&#39;</span> refs/remotes/origin <span class="p">|</span> sed <span class="s1">&#39;s#^\([0-9]*\)\torigin/#\1\t#&#39;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">apps</span><span class="o">=</span><span class="k">$(</span>fly apps list --org <span class="s2">&#34;</span><span class="nv">$ORG</span><span class="s2">&#34;</span> --json <span class="p">|</span> grep -o <span class="s1">&#39;&#34;Name&#34;: *&#34;myapp-[^&#34;]*&#34;&#39;</span> <span class="p">|</span> cut -d<span class="s1">&#39;&#34;&#39;</span> -f4<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> app in <span class="nv">$apps</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">  <span class="nv">env</span><span class="o">=</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">app</span><span class="p">#myapp-</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$env</span><span class="s2">&#34;</span> <span class="o">=</span>~ ^<span class="o">[</span>a-z0-9-<span class="o">]</span>+-<span class="o">[</span>0-9a-f<span class="o">]{</span>6<span class="o">}</span>$ <span class="o">]]</span> <span class="o">||</span> <span class="k">continue</span>       <span class="c1"># not a branch environment (e.g. myapp-preview-pg)</span>
</span></span><span class="line"><span class="cl">  grep -qx <span class="s2">&#34;</span><span class="nv">$env</span><span class="s2">&#34;</span> <span class="o">&lt;&lt;&lt;</span><span class="s2">&#34;</span><span class="nv">$live</span><span class="s2">&#34;</span> <span class="o">&amp;&amp;</span> <span class="k">continue</span>
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;sweeping </span><span class="nv">$app</span><span class="s2">: branch gone or idle &gt; </span><span class="si">${</span><span class="nv">TTL_DAYS</span><span class="si">}</span><span class="s2">d&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nv">ENV_OVERRIDE</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$env</span><span class="s2">&#34;</span> scripts/env down <span class="o">||</span> <span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span><span class="line"><span class="cl"><span class="c1"># ...then any env_* database with no app: DROP DATABASE.</span></span></span></code></pre></div></div>
<p>None of the demo branches had ever been pushed (the laptop run above), so from the sweeper&rsquo;s point of view all of them were gone, which is exactly the state a spike someone brought up from a laptop and never pushed would be in a week later. It took the remaining two apart, including the Redis keys the tagline environment had written:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ./scripts/sweep-envs
</span></span><span class="line"><span class="cl">sweeping myapp-feature-branch-envir-c65cb4: branch gone or idle &gt; 7d
</span></span><span class="line"><span class="cl">redis: deleted 0 keys under feature-branch-envir-c65cb4:*
</span></span><span class="line"><span class="cl">bucket: deleted 322 objects under feature-branch-envir-c65cb4/
</span></span><span class="line"><span class="cl">Destroyed app myapp-feature-branch-envir-c65cb4
</span></span><span class="line"><span class="cl">DROP DATABASE
</span></span><span class="line"><span class="cl">sweeping myapp-feature-profile-tagl-fa8832: branch gone or idle &gt; 7d
</span></span><span class="line"><span class="cl">redis: deleted 3 keys under feature-profile-tagl-fa8832:*
</span></span><span class="line"><span class="cl">bucket: deleted 278 objects under feature-profile-tagl-fa8832/
</span></span><span class="line"><span class="cl">Destroyed app myapp-feature-profile-tagl-fa8832
</span></span><span class="line"><span class="cl">DROP DATABASE
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ fly apps list --org myapp-preview | grep myapp-
</span></span><span class="line"><span class="cl"> myapp-preview-pg │ myapp-preview │ deployed</span></span></code></pre></div></div>
<p>(One small thing that bit: macOS ships bash 3, which has no associative arrays. The first version of the sweeper used one and died on <code>declare -A</code> on my laptop. The version above uses a newline-separated list and runs on anything.)</p>
<p>The cost is the pleasant part. A suspended machine costs its rootfs, which is cents a month. A database is 19 MB on a volume that&rsquo;s already paid for. The Redis is on a pay-as-you-go plan, twenty cents per hundred thousand commands, which at branch-testing volumes rounds to nothing. A bucket prefix costs what it stores. The shared Postgres node is the only fixed line, about two dollars a month. Twenty active branches would cost roughly what one always-on <code>shared-cpu-1x</code> costs, and the sweeper keeps it at twenty rather than two hundred. The whole demo, three environments for an hour plus the shared pieces, cost less than the coffee I drank during it.</p>

<h2 class="relative group">Migrations across branches, without the shared database to warn you
    <div id="migrations-across-branches-without-the-shared-database-to-warn-you" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#migrations-across-branches-without-the-shared-database-to-warn-you" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When each developer owns a feature end to end, every branch has a migration in it. That makes this section the important one.</p>
<p>Here&rsquo;s the thing people miss when they leave the shared-environment world. When everyone deployed to one dev database, migration conflicts surfaced immediately and painfully, but they <em>surfaced</em>. One migration ran, the next deploy failed, someone fixed it that afternoon. With a database per branch, each migration works perfectly in its own environment, and two of them first meet when the second one merges to <code>main</code>. Done carelessly, this moves the conflict from dev, where it was cheap, to a release, where it isn&rsquo;t.</p>
<p>Four rules, all enforceable by CI so nobody has to remember them.</p>
<p><strong>1. CI fails on two leaves in the migration graph, against a merge with <code>main</code>.</strong> <code>makemigrations --check</code> catches models disagreeing with migrations, but the conflict that matters is two branches both adding <code>0042_*</code>. Check it against the branch merged into <code>main</code>, not the branch alone:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git fetch origin main
</span></span><span class="line"><span class="cl">git merge --no-commit --no-ff origin/main <span class="o">||</span> <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;merge conflict&#34;</span><span class="p">;</span> <span class="nb">exit</span> 1<span class="p">;</span> <span class="o">}</span>
</span></span><span class="line"><span class="cl">python manage.py makemigrations --check --dry-run
</span></span><span class="line"><span class="cl">python manage.py migrate --plan 2&gt;<span class="p">&amp;</span><span class="m">1</span> <span class="p">|</span> grep -q <span class="s2">&#34;Conflicting migrations&#34;</span> <span class="o">&amp;&amp;</span> <span class="nb">exit</span> <span class="m">1</span></span></span></code></pre></div></div>
<p>This is the one I made happen on purpose. Two of the three demo branches, <code>feature/profile-tagline</code> and <code>feature/profile-website</code>, each added one field to the user profile model with one migration, and each environment applied its own on the way up. The new columns were only where they should be:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">env_feature_branch_envir_c65cb4      (no new column)
</span></span><span class="line"><span class="cl">env_feature_profile_tagl_fa8832      tagline
</span></span><span class="line"><span class="cl">env_feature_profile_webs_dcb2e7      website</span></span></code></pre></div></div>
<p>Both migrations are <code>0042_*</code>. Neither branch can see the other&rsquo;s, so neither environment can tell you there&rsquo;s a problem. Merging one into the other, the check does:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ git merge --no-commit --no-ff feature/profile-tagline
</span></span><span class="line"><span class="cl">$ python manage.py makemigrations --check --dry-run
</span></span><span class="line"><span class="cl">CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph:
</span></span><span class="line"><span class="cl">  (0042_profile_tagline, 0042_profile_website in accounts).
</span></span><span class="line"><span class="cl">To fix them run &#39;python manage.py makemigrations --merge&#39;
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">$ python manage.py makemigrations --merge --noinput
</span></span><span class="line"><span class="cl">Created new merge migration accounts/migrations/0043_merge_20260919_0324.py
</span></span><span class="line"><span class="cl">    dependencies = [
</span></span><span class="line"><span class="cl">        (&#39;accounts&#39;, &#39;0042_profile_tagline&#39;),
</span></span><span class="line"><span class="cl">        (&#39;accounts&#39;, &#39;0042_profile_website&#39;),
</span></span><span class="line"><span class="cl">    ]</span></span></code></pre></div></div>
<p>The second author runs that <code>--merge</code> on their branch, in their environment, bothering nobody, and the next <code>env up</code> applies <code>0043</code> on top of their clone. (In this case git also flagged a textual conflict first, because both fields had been added at the same line of the model file. Realistic, and worth knowing that the textual conflict hides the migration one until it&rsquo;s resolved.)</p>
<p><strong>2. Migrations are backwards-compatible with the previous release&rsquo;s code.</strong> During a deploy there&rsquo;s a window where the new schema and the old code coexist, or the reverse if a machine rolls back. Add columns nullable or with defaults, never rename in one step, never drop what the current release still reads: add, deploy, backfill, switch code, drop in a later release. Expand and contract. The reason it matters <em>more</em> here is that disposable environments make deploying more frequent, so you&rsquo;re inside that window more often. Staging-per-tag is where it&rsquo;s rehearsed.</p>
<p><strong>3. CI also migrates from zero into an empty database.</strong> The template clone tests &ldquo;does my migration apply to realistic data&rdquo;. The empty run tests &ldquo;does the full chain still work from nothing&rdquo;, which catches the migration that assumed a previous data migration had populated something. Thirty seconds, and it catches a class of bug that otherwise waits for the next new laptop.</p>
<p><strong>4. Schema changes get a second pair of eyes.</strong> In a small team this doesn&rsquo;t need a <code>CODEOWNERS</code> file, but it does need to be a rule: a PR that touches <code>migrations/</code> is reviewed by a teammate before merge, and the reviewer&rsquo;s job isn&rsquo;t to check the SQL. It&rsquo;s to know that they, or someone else on the team, are touching the same table this fortnight and say so. Past a dozen developers that knowledge stops fitting in one head, and a <code>CODEOWNERS</code> entry on <code>migrations/</code> earns its place. No tooling replaces this. With the tooling catching everything mechanical, it&rsquo;s a two-minute job, and it&rsquo;s the one part of the old shared-environment world worth keeping: the moment where two people find out they&rsquo;re about to collide.</p>

<h2 class="relative group">Versioning, so a URL tells you what it&rsquo;s running
    <div id="versioning-so-a-url-tells-you-what-its-running" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#versioning-so-a-url-tells-you-what-its-running" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>With one shared environment, &ldquo;what&rsquo;s on staging&rdquo; was a chat question. With many, it has to be a property of the environment or every bug report starts with archaeology.</p>
<ul>
<li><strong>Every image carries the commit SHA and the app exposes it.</strong> <code>fly deploy --build-arg GIT_SHA=... --build-arg GIT_BRANCH=...</code> stamps them into the image as environment variables, and <code>/api/version/</code> returns them with the environment name and the region (see the output above). The first thing anyone does with &ldquo;it&rsquo;s broken on feature-profile-tagl-fa8832&rdquo; is check it&rsquo;s running the commit they think. The first demo environment was deployed from the working tree a commit before the tooling was committed, and the endpoint said so: it reported the previous commit, not the SHA I expected.</li>
<li><strong><code>ENVIRONMENT=$ENV</code> is in every log line and the error tracker&rsquo;s environment tag.</strong> Branch errors don&rsquo;t pollute prod&rsquo;s error tracking and are filterable in one click.</li>
<li><strong>Branches get SHAs, releases get tags.</strong> <code>feature-payments-retr-3f9a1c</code> never has a version number; it has commits. Staging and prod have <code>v1.42.0</code>.</li>
<li><strong>The mobile app carries a minimum API version, not a URL.</strong> The dev-menu URL field is how a tester points a real phone at a branch, and the version endpoint carrying an API schema version is how the app knows whether it&rsquo;s talking to something it understands.</li>
</ul>

<h2 class="relative group">What this doesn&rsquo;t solve
    <div id="what-this-doesnt-solve" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-this-doesnt-solve" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>It doesn&rsquo;t solve two features that only make sense together. A branch environment shows one branch. The answer is to merge both behind flags and look at <code>main</code>, which you can do with one standing environment that tracks <code>main</code> (<code>main-dev.example.com</code>, redeployed on every merge). That&rsquo;s the one long-lived non-prod environment I&rsquo;d keep, and nobody deploys <em>to</em> it; it&rsquo;s for seeing what the next release will be.</p>
<p>It doesn&rsquo;t solve providers that want a registered callback and don&rsquo;t accept wildcards. My auth provider does accept a parent domain, so <code>example.com</code> covers every <code>-dev</code> branch hostname. Some OAuth and payment providers don&rsquo;t, and for those you keep one shared hostname that the Worker routes to whichever environment last claimed it. That&rsquo;s a small lock over a small thing, and it&rsquo;s a lot better than a lock over everything.</p>
<p>It doesn&rsquo;t solve the human side of migrations. The tooling catches mechanical conflicts. It doesn&rsquo;t catch two of you modelling the same concept incompatibly in the same fortnight, and the only fix for that is the migration review where one of you notices.</p>

<h2 class="relative group">What I&rsquo;d check in your setup
    <div id="what-id-check-in-your-setup" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-check-in-your-setup" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you have a shared dev or staging and a chat thread for booking it:</p>
<ol>
<li>How long does a finished branch wait before someone other than its author can see it running? More than an hour and you&rsquo;re paying for the lock in delivery time, and on a small team that&rsquo;s a big fraction of it waiting.</li>
<li>How often does the shared database get reset, and who loses work when it does?</li>
<li>Could a new developer, on day one, get a public URL running their branch without asking anyone for anything? Whatever they&rsquo;d have to ask for is the thing to automate first.</li>
<li>Does a fresh environment start from realistic data? If it starts empty, the first bug it won&rsquo;t catch is the migration that&rsquo;s fine on an empty table and takes forty minutes on the real one.</li>
<li>Is there anything in your non-prod hosting that would still exist a month after everyone forgot about it? If so, write the sweeper before anything else. The leak starts on day one.</li>
</ol>
<p>The chat question goes away not because people stop asking it but because the answer is always &ldquo;you are, on yours.&rdquo;</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-09-19-ephemeral-dev-environments-on-fly/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Django on Fly.io: Load Testing and Autoscaling on Minimal Compute</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/</guid>
      <pubDate>Wed, 22 Jul 2026 00:00:00 +0000</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Django</category>
      <category>Python</category>
      <category>React Native</category>
      <category>Expo</category>
      <category>Fly.io</category>
      <category>PostgreSQL</category>
      <category>WebSockets</category>
      <category>Scaling</category>
      <category>Load Testing</category>
      <category>Performance</category>
      <category>Tech Stack</category>
      <description>How I run two Django apps on Fly.io&#39;s shared-cpu-1x machines: make one machine fast, load-test it to find where latency collapses, and set Fly.io autoscaling just below it.</description>
      <content:encoded><![CDATA[<p>To run Django on Fly.io cheaply, I make one small shared-cpu-1x machine as fast as I can, load-test it until latency collapses, and set Fly&rsquo;s autoscaler to start a second machine just before that point. For the Django load testing I use Locust with a step-load ramp (and, on the other app, a small Python script that sweeps concurrency), watching p95 latency and error rates. When traffic drops, Fly stops the idle machines again, so the baseline cost is one small VM per app.</p>
<p>I&rsquo;m currently shipping two products as a solo developer: a <strong>field estimation app</strong> for tradespeople who quote jobs on site, and a <strong>marketplace and community app</strong> with listings and real-time messaging.</p>
<p>Different products, different users, nearly identical stack — and both run entirely on <strong>shared-cpu-1x machines</strong>, the cheapest tier Fly.io offers. That constraint is deliberate. Instead of sizing servers for peak load, I squeeze as much as possible out of one small machine, measure exactly where it collapses, and let Fly&rsquo;s autoscaler add and remove machines around that measured limit. This post covers the optimisations that make that possible, the load-test numbers behind the thresholds, and how scaling up and down actually works on Fly.io.</p>

<h2 class="relative group">The Philosophy: Measure One Machine, Then Multiply
    <div id="the-philosophy-measure-one-machine-then-multiply" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-philosophy-measure-one-machine-then-multiply" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The scaling strategy for both apps is the same three steps:</p>
<ol>
<li>Make a single small machine as fast as it can reasonably be.</li>
<li>Load-test it until it breaks, and note the concurrency level where latency collapses.</li>
<li>Configure Fly&rsquo;s autoscaler to start a second machine just before that point, and stop idle machines when traffic drops.</li>
</ol>
<p>Capacity then scales roughly linearly with machine count, and the baseline cost is one small VM per app. Everything below serves one of those three steps.</p>

<h2 class="relative group">Making One Small Machine Fast
    <div id="making-one-small-machine-fast" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#making-one-small-machine-fast" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">Granian: WSGI for the API, uvloop ASGI for WebSockets
    <div id="granian-wsgi-for-the-api-uvloop-asgi-for-websockets" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#granian-wsgi-for-the-api-uvloop-asgi-for-websockets" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Both backends are Django 6 + Django REST Framework. The API is served by <strong>Granian</strong>, a Rust-based server — but under <strong>WSGI with a blocking thread pool</strong>, not ASGI:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">granian --interface wsgi myapp.wsgi:application <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers <span class="m">1</span> --blocking-threads <span class="m">16</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers-max-rss <span class="m">1400</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers-kill-timeout <span class="m">15</span> --respawn-failed-workers</span></span></code></pre></div></div>
<p>I started on <code>--interface asgi --loop uvloop</code>, and it was the wrong call. Every HTTP view in these apps is sync, and Django&rsquo;s ASGI handler runs sync views <code>thread_sensitive</code> — one at a time per worker. So ASGI gave me roughly one concurrent request per worker <em>and</em> loaded the whole asyncio/uvloop/Channels stack into RAM for nothing. WSGI with <code>--blocking-threads</code> runs those same sync views in a real thread pool: one worker × 16 threads = 16 concurrent I/O-bound requests (DB reads, S3 presigns) at lower resident memory than the async build.</p>
<p>Why a single worker? The VM is <code>shared-cpu-1x</code> — one core. A second worker can&rsquo;t run Python in parallel on one core (GIL + one CPU), but it does cost a whole extra resident Django process; Granian warns about exactly this. Sixteen threads on one worker is the honest ceiling of a 1-vCPU box for I/O-bound work. <code>--workers-max-rss 1400</code> respawns the worker if it drifts past its memory cap, so a runaway PDF or image job gets recycled instead of walking the machine toward OOM.</p>
<p>WebSockets moved out to their own Granian process — an <strong>ASGI</strong> app with uvloop and <code>--ws</code>, one worker for long-lived connections. The sync API needs no ASGI path at all: when a view wants to push to a socket it calls <code>async_to_sync(channel_layer.group_send)</code> over Redis, so the message still crosses to the WebSocket process. Two servers, each running the interface it&rsquo;s actually good at. Local Docker runs the same WSGI thread pool, so my laptop approximates one production machine when testing.</p>

<h3 class="relative group">Rust in the Hot Paths
    <div id="rust-in-the-hot-paths" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rust-in-the-hot-paths" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Wherever a Rust-backed drop-in exists for a Python hot path, I take it:</p>
<ul>
<li><strong>orjson</strong> via <code>drf-orjson-renderer</code> — every API response serialises through Rust instead of the stdlib JSON encoder.</li>
<li><strong>blurhash-rs</strong> — image placeholders computed at upload time.</li>
<li><strong>uuid-utils</strong> — Rust UUID generation.</li>
<li><strong>psycopg 3 (binary)</strong> — C-accelerated Postgres driver.</li>
</ul>
<p>None of these change the architecture; they just raise the ceiling of what one vCPU can do before you need a second machine.</p>

<h3 class="relative group">Kill the Hidden Latency Taxes
    <div id="kill-the-hidden-latency-taxes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#kill-the-hidden-latency-taxes" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The single biggest win in either codebase wasn&rsquo;t an algorithm — it was connection reuse. Under Granian&rsquo;s thread pool, every new thread was building a fresh boto3 S3 client, and every endpoint that returns a photo or PDF paid that cost. A small custom storage class now caches the client once per process instead:</p>
<blockquote><p>Signed URLs went from <strong>~150ms to ~1ms</strong> on new threads.</p>
</blockquote><p>The same lesson applies to the database: Fly&rsquo;s Postgres sits behind PgBouncer, so the app runs <code>conn_max_age=0</code> and lets the pooler own connection reuse, with a 5-second <code>connect_timeout</code> to fail fast on stale SSL instead of hanging a worker.</p>

<h3 class="relative group">Cache Where the Bottleneck Actually Is
    <div id="cache-where-the-bottleneck-actually-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cache-where-the-bottleneck-actually-is" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Load testing the marketplace revealed something worth internalising: <strong>the ceiling was Postgres connections, not CPU</strong>. Browse traffic is overwhelmingly anonymous and identical across users, and under load those identical list queries saturate the connection pool while the CPU sits idle.</p>
<p>The fix is a Redis cache for anonymous list pages, with per-namespace version counters for invalidation — a write bumps one counter (O(1), no key bookkeeping) and every cached page for that resource is instantly stale. A short TTL backstops anything missed. Serving browse pages straight from Redis removes the real bottleneck and lets the small machine spend its connections on logged-in traffic.</p>

<h3 class="relative group">Background Work Without a Broker
    <div id="background-work-without-a-broker" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#background-work-without-a-broker" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Both apps use <strong>django-tasks-db</strong>, Django&rsquo;s database-backed task framework, instead of Celery. A separate <code>db_worker</code> process polls the database for jobs — image processing, PDF generation, machine translation, reminders. No broker, no extra service to size and monitor, and the API process never blocks on heavy work. The worker even connects straight to Postgres, bypassing PgBouncer, since long-running jobs don&rsquo;t play well with transaction pooling.</p>

<h3 class="relative group">Memory: jemalloc and a Swap Cushion
    <div id="memory-jemalloc-and-a-swap-cushion" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#memory-jemalloc-and-a-swap-cushion" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Image and PDF processing spike memory in ways steady-state sizing doesn&rsquo;t predict. Two cheap mitigations: <strong>jemalloc</strong> preloaded in the Docker image to reduce fragmentation, and <strong>swap on every machine</strong> (2GB on the API machines, 1GB on the WebSocket machine) so a spike swaps to disk instead of getting the process OOM-killed. Swap is not a performance feature — it&rsquo;s an insurance policy that lets me keep memory allocations small.</p>

<h2 class="relative group">Django Load Testing: Finding the Knee
    <div id="django-load-testing-finding-the-knee" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#django-load-testing-finding-the-knee" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Thresholds are only defensible if they&rsquo;re measured. Each project has its own harness:</p>
<ul>
<li>The estimation app has a small Python stress script that sweeps concurrency levels and reports RPS, p50/p95/p99 and error rates. It runs against a few hundred generated sample records, so the list endpoints are queried at realistic data volumes.</li>
<li>The marketplace uses <strong>Locust</strong> with a step-load shape that ramps 50 → 500 users in 30-second steps to find the saturation point, with endpoint weights matching real browse-heavy traffic, plus a second, smaller suite that ramps 5 → 50 virtual users against production with a p95 &lt; 2s threshold.</li>
</ul>
<p>Representative numbers from the estimation app&rsquo;s sweep (one Granian worker × 16 blocking threads, Docker on my dev machine):</p>
<table>
	<thead>
			<tr>
					<th>Endpoint</th>
					<th>Concurrency</th>
					<th>RPS</th>
					<th>p95</th>
					<th>Errors</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>health</td>
					<td>25</td>
					<td>602</td>
					<td>52ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>heavy list</td>
					<td>25</td>
					<td>309</td>
					<td>260ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>light list</td>
					<td>25</td>
					<td>357</td>
					<td>61ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>heavy list</td>
					<td>50</td>
					<td>309</td>
					<td>347ms</td>
					<td>0%</td>
			</tr>
	</tbody>
</table>
<p>And the number that drives everything else comes from load-testing the marketplace on a <strong>single shared-cpu-1x machine</strong>. On the old ASGI build this was a hard wall: sync views serialised to roughly <strong>three usable slots</strong>, so p95 held under 300ms up to ~5 concurrent requests and then collapsed past <strong>~8–10</strong> as everything queued behind the thread-sensitive lock. Moving the API to a WSGI thread pool is what pushed that knee out — <strong>16 real concurrent slots</strong> per machine instead of ~3 — and that new ceiling is the whole basis of the autoscaling config.</p>

<h2 class="relative group">How Scaling Works on Fly.io
    <div id="how-scaling-works-on-flyio" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-scaling-works-on-flyio" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly&rsquo;s model is simple: your app is a fleet of identical machines behind a proxy, and the proxy counts load per machine. You declare two numbers:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">concurrency</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s1">&#39;requests&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">soft_limit</span> <span class="p">=</span> <span class="mi">12</span>   <span class="c"># past this, start another machine</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hard_limit</span> <span class="p">=</span> <span class="mi">24</span>   <span class="c"># past this, shed load instead of queueing</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;stop&#39;</span>
</span></span><span class="line"><span class="cl"><span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl"><span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p><strong>Scaling up:</strong> when in-flight requests on a machine exceed the soft limit, the proxy starts a stopped machine and routes new traffic there. The WSGI thread pool gives each machine 16 real concurrent slots, so the soft limit sits at 12 — about 75% of the pool, so machine #2 starts while threads still have headroom rather than after users are seeing multi-second latency. The hard limit of 24 (1.5× the pool) is the circuit breaker: short bursts queue on threads, and beyond it the proxy sheds load rather than letting one machine melt down. (The old ASGI build ran 8/20, sized for its ~3 real slots — widening the pool is what let both numbers move up.)</p>
<p><strong>Scaling down:</strong> when traffic drops, Fly stops idle machines automatically, down to <code>min_machines_running = 1</code>. A stopped machine costs nothing for compute, only its rootfs storage, so the baseline cost of each app is literally one small VM with burst capacity always ready.</p>
<p>I originally wrote here that a stopped machine &ldquo;cold-starts in well under a second.&rdquo; That was wrong, and expensively so. When I finally measured it instead of assuming it, a cold start took <strong>25 to 33 seconds</strong>, and a request that lands on a machine still booting has to wait out all of it. I first blamed those cold starts for a 22% failure rate on my CDN dashboard, while every health check stayed green. That number turned out to be the CDN&rsquo;s own Early Hints probes failing against the origin; no real client got a 504. The cold starts were real all the same.</p>
<p>The fix is not in the scaling numbers above; those are fine. It is in what the machine does on the way up, and in asking Fly to suspend machines rather than stop them, which turns a wake into a ~3 second memory restore. I wrote that up separately: <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >Scale to Zero, and Every Restart Is a 30-Second Outage</a> covers the diagnosis and what to change. If you are running anything on Fly with <code>auto_stop_machines</code>, read that one before you trust this paragraph.</p>
<p>One subtlety: the concurrency type is <code>requests</code>, not <code>connections</code>. Modern clients hold keep-alive connections open, so counting connections barely moves under real load and the autoscaler never triggers. Counting in-flight requests tracks actual work.</p>

<h3 class="relative group">WebSockets Scale on a Different Axis
    <div id="websockets-scale-on-a-different-axis" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#websockets-scale-on-a-different-axis" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The marketplace splits its WebSocket server into a <strong>second Fly app</strong> sharing the same Docker image. The API app runs one worker × 16 blocking threads under WSGI and counts <em>requests</em> (soft limit 12). The WebSocket app runs 1 worker under ASGI with uvloop on a <strong>512MB</strong> machine and counts <em>connections</em>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">concurrency</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s1">&#39;connections&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">soft_limit</span> <span class="p">=</span> <span class="mi">1500</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hard_limit</span> <span class="p">=</span> <span class="mi">2000</span></span></span></code></pre></div></div>
<p>A long-lived socket is nearly free to hold open but occupies a slot for hours, while an HTTP request occupies a slot for milliseconds — the same autoscaling signal can&rsquo;t serve both. Splitting them means each fleet scales on its own honest metric, and idle WebSocket capacity costs 512MB, not 2GB. On the Channels side, the Redis layer runs with <code>capacity: 1500</code> and 60-second message expiry, plus socket keepalives every 30 seconds because Fly&rsquo;s Redis proxy idle-closes the long-lived BRPOP sockets that Channels depends on.</p>

<h2 class="relative group">Offloading Compute to the Edge and the Client
    <div id="offloading-compute-to-the-edge-and-the-client" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#offloading-compute-to-the-edge-and-the-client" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The cheapest server work is the work the server never does:</p>
<ul>
<li><strong>Static and media files</strong> live in Fly&rsquo;s S3-compatible object storage, served through a CDN with signed URLs for private media — zero image bytes transit the Django machines.</li>
<li><strong>AI inference runs on the user&rsquo;s phone.</strong> Both apps embed a small local LLM (a quantised 4B model, around 2.7GB) that runs on the device. Zero inference cost server-side, and it works offline.</li>
<li><strong>Offline-first mobile</strong> (SQLite on the device and a sync queue with retries) means the apps batch writes and tolerate dead zones instead of hammering the API with chatty per-keystroke requests.</li>
</ul>

<h2 class="relative group">The Rest of the Stack, Briefly
    <div id="the-rest-of-the-stack-briefly" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-rest-of-the-stack-briefly" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Auth is a hosted identity provider plus WebAuthn passkeys, and WebSocket connections are authenticated before the upgrade. Postgres in production, SQLite in development, and a couple of off-the-shelf Django packages for the admin and audit history. Mobile is React Native and TypeScript on Expo, with Expo Router and Tailwind, over-the-air updates from a self-hosted Expo Updates server, and a subscriptions service for in-app purchases. Quality gates: pytest and mypy on the backend, unit and end-to-end tests on the client.</p>

<h2 class="relative group">What I&rsquo;d Tell You to Steal
    <div id="what-id-tell-you-to-steal" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-tell-you-to-steal" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The transferable idea isn&rsquo;t any single library — it&rsquo;s the loop. Make one cheap machine genuinely fast (Rust hot paths, connection reuse, cache in front of the real bottleneck, offload what you can to CDN and client). Load-test until you find the exact concurrency where it breaks. Set the autoscaler&rsquo;s soft limit at that knee, let idle machines stop, and keep the minimum fleet at one. Two production apps, real-time features, AI assistants — and the steady-state bill is a couple of the cheapest VMs on the platform.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
