<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Cold Starts · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/cold-starts/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Sat, 07 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/cold-starts/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Fly.io Cold Starts with Scale to Zero: From 33 Seconds to 3</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/</guid>
      <pubDate>Wed, 09 Sep 2026 00:00:00 +1200</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Fly.io</category>
      <category>Django</category>
      <category>DevOps</category>
      <category>Cloudflare</category>
      <category>Performance</category>
      <category>Cold Starts</category>
      <category>Redis</category>
      <category>Debugging</category>
      <description>Why my Fly.io machines took 25 to 33 seconds to cold start with auto_stop_machines, and how a release lock, compiled bytecode and suspend got wake-ups down to about 3 seconds.</description>
      <content:encoded><![CDATA[<p><strong>Update, 22 October 2026:</strong> the headline number in this post is wrong. On the day I published it I grouped the zone&rsquo;s 504s by user agent, and all 7,344 of them between 5 and 9 September came from Cloudflare&rsquo;s own Early Hints probes (<code>bastion early hints</code> and <code>nginx-ssl early hints</code>). Those get no response from the Fly origin, are logged as 504s, and mirror real request paths, so they look exactly like user failures. No real client (the app, browsers, curl, search crawlers) got a 504 in that window. I turned Early Hints off the same day: the origin sends no preload headers, so it was doing nothing useful. The cold starts below were real, and so are the fixes, but the 22% was never what my users saw. Before you treat a Cloudflare 5xx rate as an outage, check which user agents are getting them.</p>
<p><strong>Update, 4 November 2026:</strong> one more correction. Those 504s aren&rsquo;t probes the origin fails to answer. They&rsquo;re Cloudflare&rsquo;s own Early Hints cache lookups, which never reach the origin at all, and a 504 there only means nothing was cached for that URL. Turning Early Hints off wasn&rsquo;t the best fix either: sending a <code>Link</code> response header made the feature work and turned those misses into hits. The full story is in <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >Cloudflare 504 errors that aren&rsquo;t real: Early Hints and HTTP 103</a>.</p>
<p>With <code>auto_stop_machines</code> on, any request that lands on a stopped Fly.io machine waits for a full boot, and mine took 25 to 33 seconds because the entrypoint ran Django migrations and <code>collectstatic</code> on every machine start, and nothing had ever cached the bytecode. Running that release work once per image (a Redis lock keyed on <code>FLY_IMAGE_REF</code>) and adding <code>compileall</code> to the build got a cold boot down to about nine seconds. Switching <code>auto_stop_machines</code> from <code>'stop'</code> to <code>'suspend'</code>, which meant dropping a swap file Fly won&rsquo;t suspend with, got wake-ups to about three.</p>
<p>I went looking for a traffic spike on my Cloudflare dashboard and found something much worse underneath it: 22% of every request my app served in a day had failed. No alert had fired. Every health check was green. The logs looked perfectly healthy, because the failing requests never reached the application at all.</p>
<p>The cause turned out to be my own container startup script, doing sensible work at exactly the wrong time.</p>

<h2 class="relative group">The spike was a red herring
    <div id="the-spike-was-a-red-herring" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-spike-was-a-red-herring" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The thing I set out to explain was one hour that took about 4,900 requests against a median of 121. Forty-two unique visitors. That is 110 requests per visitor, which is not how humans browse.</p>
<p>It was a scanner. A single bot walking through invented subdomains, asking each one for secrets:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">api-stage-asia.example.com/.env.local
</span></span><span class="line"><span class="cl">api-us-east-1-demo.example.com/.env
</span></span><span class="line"><span class="cl">api-eu-uat.example.com/wp-config.php~
</span></span><span class="line"><span class="cl">app-test-us-east-1.example.com/actuator/configprops
</span></span><span class="line"><span class="cl">prod-api-use2.example.com/.env.local
</span></span><span class="line"><span class="cl">relay.example.com/.env.production</span></span></code></pre></div></div>
<p>Thirty or so hostnames, every <code>.env</code> variant you can think of, plus a Java framework&rsquo;s actuator endpoints on a Django app. All of them got a 301 to my canonical host and nothing was served. Boring, and genuinely harmless.</p>
<p>But while I had the analytics open, I grouped the same day by status code. That is where the real problem was.</p>
<table>
	<thead>
			<tr>
					<th>Status</th>
					<th>Requests in 24h</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Total</td>
					<td>11,350</td>
			</tr>
			<tr>
					<td>504 Gateway Timeout</td>
					<td>2,474</td>
			</tr>
	</tbody>
</table>
<p>Twenty-two percent. And not spread evenly across the site, but concentrated exactly where it hurts:</p>
<table>
	<thead>
			<tr>
					<th>Endpoint</th>
					<th>504s</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>notifications unread count</td>
					<td>621</td>
			</tr>
			<tr>
					<td>notifications list</td>
					<td>614</td>
			</tr>
			<tr>
					<td>messages unread count</td>
					<td>469</td>
			</tr>
			<tr>
					<td>feed cursor</td>
					<td>210</td>
			</tr>
	</tbody>
</table>
<p>Those are the endpoints my mobile app polls every 30 seconds. So this was not an abstract error-rate number. It was every user&rsquo;s phone, quietly failing to load their unread badge, all day, for weeks probably.</p>

<h2 class="relative group">Why the logs looked fine
    <div id="why-the-logs-looked-fine" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-the-logs-looked-fine" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My first instinct was that the app had become slow. It had not. I pulled the application&rsquo;s own request lines for one of the worst hours out of the log archive:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[2026-09-08 01:00:29] &#34;GET /api/notifications/unread/&#34; 304 61.615
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:29] &#34;GET /api/notifications/&#34;        304 63.568
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:59] &#34;GET /api/notifications/unread/&#34; 304 58.742
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:59] &#34;GET /api/notifications/&#34;        304 105.438</span></span></code></pre></div></div>
<p>Forty-four to a hundred milliseconds, 304s all the way down. The origin was fast.</p>
<p>That is the important clue, and it is worth stating as a rule: <strong>if your edge reports errors your origin has never heard of, the requests are dying before they arrive.</strong> Stop reading application logs and start looking at what sits in front of them.</p>
<p>In my case that is Fly.io&rsquo;s proxy, and the answer was in how my machines are scheduled.</p>

<h2 class="relative group">Scale to zero is a promise about money, not latency
    <div id="scale-to-zero-is-a-promise-about-money-not-latency" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#scale-to-zero-is-a-promise-about-money-not-latency" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My config looks like this, and on the face of it it is reasonable:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;stop&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">  <span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p>Idle machines stop. Traffic starts them again. You pay for what you use. For a small app with bursty traffic this is exactly what you want, and it is why I had ten machines defined but almost always one running.</p>
<p>The bill it hides is latency. When a request arrives for a stopped machine, somebody has to wait for that machine to boot. If booting takes longer than the proxy&rsquo;s patience, that somebody gets a 504.</p>
<p>So: how long did my machines take to boot? I had never measured. The log archive knows, because the startup script announces each phase, and every line carries the machine id:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">WITH</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">SELECT</span><span class="w"> </span><span class="n">instance</span><span class="p">,</span><span class="w"> </span><span class="k">timestamp</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">         </span><span class="k">CASE</span><span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Starting app server%&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;begin&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Migrations complete%&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;migrated&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">LIKE</span><span class="w">  </span><span class="s1">&#39;%&#34;GET /healthz%&#39;</span><span class="w">       </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;serving&#39;</span><span class="w"> </span><span class="k">END</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">phase</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">WHERE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;myapp&#39;</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="k">timestamp</span><span class="p">::</span><span class="nb">date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;2026-09-08&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="n">instance</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;begin&#39;</span><span class="p">)</span><span class="w">    </span><span class="k">AS</span><span class="w"> </span><span class="n">started</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;migrated&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">migrated</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;serving&#39;</span><span class="p">)</span><span class="w">  </span><span class="k">AS</span><span class="w"> </span><span class="n">first_serve</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">instance</span></span></span></code></pre></div></div>
<p>The answer, consistently, across every machine:</p>
<table>
	<thead>
			<tr>
					<th>Phase</th>
					<th>Duration</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Boot to migrations finished</td>
					<td>10 to 13 seconds</td>
			</tr>
			<tr>
					<td>Migrations to first served request</td>
					<td>8 to 19 seconds</td>
			</tr>
			<tr>
					<td><strong>Total</strong></td>
					<td><strong>25 to 33 seconds</strong></td>
			</tr>
	</tbody>
</table>
<p>Then I counted how often that happened:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">starts</span><span class="p">,</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="k">DISTINCT</span><span class="w"> </span><span class="n">instance</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">machines</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">WHERE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;myapp&#39;</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="k">timestamp</span><span class="p">::</span><span class="nb">date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;2026-09-08&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">AND</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Starting app server%&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">-- starts: 92, machines: 10</span></span></span></code></pre></div></div>
<p>Ninety-two cold starts in one day. Each one a half-minute window in which anything routed to that machine timed out. Twenty-two percent stopped looking mysterious.</p>

<h2 class="relative group">The cause was in my own entrypoint
    <div id="the-cause-was-in-my-own-entrypoint" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-cause-was-in-my-own-entrypoint" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Here is what every single machine ran before it would answer a request:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># entrypoint.sh, run on every machine start</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Running migrations...&#34;</span>
</span></span><span class="line"><span class="cl">python manage.py migrate --noinput
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Running collectstatic in background...&#34;</span>
</span></span><span class="line"><span class="cl">python manage.py collectstatic --noinput <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">exec</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span>    <span class="c1"># then start the app server</span></span></span></code></pre></div></div>
<p>Read that again with &ldquo;92 times a day&rdquo; in mind.</p>
<p><code>migrate</code> is a full Django boot: import every model, connect to Postgres, query the migrations table, decide there is nothing to do. On a shared CPU that is ten seconds to accomplish nothing. <code>collectstatic</code> is a second full Django boot that then uploads static files to object storage, competing for the same single core that the web server is trying to start on. The app server is the third boot.</p>
<p>Three Python interpreters, two of them doing work that had already been done, every time a machine woke up.</p>
<p>The reason it was written that way is good, and it is probably why yours looks similar: migrations have to run somewhere, Fly&rsquo;s <code>release_command</code> had hung on me before, and the entrypoint is the one place guaranteed to execute on deploy. It is a correct place to put deploy work. It is just not a correct place to put it <em>per machine start</em>, and once your machines stop and start on their own, those are no longer the same event.</p>
<p>That is the whole bug, and I think it is a common one:</p>
<blockquote><p>Work that belongs to a <strong>release</strong> was being run once per <strong>machine start</strong>. Scale-to-zero turned one event into ninety-two.</p>
</blockquote>
<h2 class="relative group">The fix: claim the release once
    <div id="the-fix-claim-the-release-once" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-fix-claim-the-release-once" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Migrations do not need to run on the eighth machine to boot a given image. They need to run once for that image, and everyone else needs to know it has happened.</p>
<p>That is a lock with a name that changes per deploy. Fly hands you the name for free in <code>FLY_IMAGE_REF</code>, and I already had Redis. The whole thing is a small script that runs before Django exists, so the common answer costs milliseconds rather than a framework boot:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># release_gate.py: runs before Django exists. Exit code 0 means &#34;prepare this release&#34;.</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">PREPARE</span><span class="p">,</span> <span class="n">SKIP</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl"><span class="n">LOCK_TTL</span><span class="p">,</span> <span class="n">WAIT</span><span class="p">,</span> <span class="n">POLL</span> <span class="o">=</span> <span class="mi">300</span><span class="p">,</span> <span class="mi">240</span><span class="p">,</span> <span class="mi">2</span>
</span></span><span class="line"><span class="cl"><span class="n">_sleep</span><span class="p">,</span> <span class="n">_monotonic</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span>    <span class="c1"># aliases, so tests can replace them</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">release_id</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Something that changes exactly when the deployed image does.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_IMAGE_REF&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_MACHINE_VERSION&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">claim</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">release</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">release</span><span class="p">:</span>   <span class="c1"># no Redis, or no idea which release: behave as before</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">PREPARE</span>
</span></span><span class="line"><span class="cl">    <span class="n">done_key</span><span class="p">,</span> <span class="n">lock_key</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;release:</span><span class="si">{</span><span class="n">release</span><span class="si">}</span><span class="s2">:done&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;release:</span><span class="si">{</span><span class="n">release</span><span class="si">}</span><span class="s2">:lock&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">done_key</span><span class="p">):</span>            <span class="c1"># somebody already did it for this image</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">SKIP</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">owner</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_MACHINE_ID&#34;</span><span class="p">,</span> <span class="s2">&#34;unknown&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="n">LOCK_TTL</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">PREPARE</span>                  <span class="c1"># we won the race; we do the work</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Someone else is preparing. Wait for them, because serving requests</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># against a half-migrated schema is worse than starting slowly.</span>
</span></span><span class="line"><span class="cl">    <span class="n">deadline</span> <span class="o">=</span> <span class="n">_monotonic</span><span class="p">()</span> <span class="o">+</span> <span class="n">WAIT</span>
</span></span><span class="line"><span class="cl">    <span class="k">while</span> <span class="n">_monotonic</span><span class="p">()</span> <span class="o">&lt;</span> <span class="n">deadline</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">_sleep</span><span class="p">(</span><span class="n">POLL</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">done_key</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">SKIP</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="n">LOCK_TTL</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">PREPARE</span>              <span class="c1"># the owner died and its lock expired; take over</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">PREPARE</span>                      <span class="c1"># waited long enough; do it ourselves</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># ... plus `done` (sets done_key) and a main() that treats any Redis error as PREPARE</span></span></span></code></pre></div></div>
<p>The entrypoint becomes an <code>if</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="k">if</span> python3 release_gate.py claim<span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    python manage.py migrate --noinput
</span></span><span class="line"><span class="cl">    python manage.py collectstatic --noinput <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">    python3 release_gate.py <span class="k">done</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">exec</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span></span></span></code></pre></div></div>

<h2 class="relative group">Every failure has to end in &ldquo;prepare it anyway&rdquo;
    <div id="every-failure-has-to-end-in-prepare-it-anyway" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-failure-has-to-end-in-prepare-it-anyway" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is the part worth copying, more than the code.</p>
<p>A gate like this sits between your users and a schema migration. If it is wrong in the direction of <em>skipping</em>, you serve requests against a database that has not been migrated, which is a genuinely bad outage. If it is wrong in the direction of <em>preparing</em>, you run an idempotent no-op migration twice and waste ten seconds.</p>
<p>Those outcomes are not symmetrical, so the code must not treat them as such. Every uncertain path returns <code>PREPARE</code>:</p>
<ul>
<li><strong>No Redis, or Redis unreachable.</strong> Prepare. This is exactly the old behaviour, so a cache outage degrades to &ldquo;slow&rdquo;, not &ldquo;broken&rdquo;.</li>
<li><strong>No image reference in the environment.</strong> Prepare. I do not know what release this is, so I cannot know whether it is ready.</li>
<li><strong>The lock&rsquo;s owner died mid-migration.</strong> Its lock expires, the next waiter takes it over and prepares.</li>
<li><strong>Waited longer than the timeout.</strong> Prepare anyway. A machine that never serves is worse than a duplicated migration.</li>
</ul>
<p>There is one deliberate asymmetry in the other direction: a machine that finds another machine actively holding the lock <em>waits</em> rather than serving. Starting slowly during a deploy is fine. Answering queries against a half-applied schema is not.</p>
<p>Watch out for one trap if you build this on a cache wrapper that swallows connection errors, as I first did. A wrapper like that returns <code>False</code> from <code>add()</code> when Redis is unreachable, which reads identically to &ldquo;someone else holds the lock&rdquo; and would have made every machine skip its migrations. That is precisely the dangerous direction. Distinguish the two by reading the lock back: a lock nobody holds means the cache is broken, not that you lost a race.</p>

<h2 class="relative group">Testing it without a Redis
    <div id="testing-it-without-a-redis" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#testing-it-without-a-redis" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The logic is worth unit testing, because the interesting paths are the ones you cannot reproduce by hand. A fake with two methods covers it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FakeRedis</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Enough of a Redis client for the gate: get, and set with nx/ex.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">nx</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">True</span></span></span></code></pre></div></div>
<p>Then drive the awkward cases by making time pass on your terms. This is why the gate calls module-level <code>_sleep</code> and <code>_monotonic</code> aliases rather than <code>time.sleep</code> directly, so a test can replace them without patching the <code>time</code> module for every other thread in the process:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_a_waiter_takes_over_when_the_owner_disappears</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">redis</span> <span class="o">=</span> <span class="n">FakeRedis</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">release_gate</span><span class="o">.</span><span class="n">claim</span><span class="p">(</span><span class="n">redis</span><span class="p">,</span> <span class="s2">&#34;img-1&#34;</span><span class="p">),</span> <span class="n">release_gate</span><span class="o">.</span><span class="n">PREPARE</span><span class="p">)</span>   <span class="c1"># the owner takes the lock</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">owner_dies</span><span class="p">(</span><span class="n">_seconds</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">redis</span><span class="o">.</span><span class="n">store</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s2">&#34;release:img-1:lock&#34;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>                              <span class="c1"># its lock expired</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">patch</span><span class="o">.</span><span class="n">object</span><span class="p">(</span><span class="n">release_gate</span><span class="p">,</span> <span class="s2">&#34;_sleep&#34;</span><span class="p">,</span> <span class="n">owner_dies</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">release_gate</span><span class="o">.</span><span class="n">claim</span><span class="p">(</span><span class="n">redis</span><span class="p">,</span> <span class="s2">&#34;img-1&#34;</span><span class="p">),</span> <span class="n">release_gate</span><span class="o">.</span><span class="n">PREPARE</span><span class="p">)</span></span></span></code></pre></div></div>

<h2 class="relative group">Did it work? Partly
    <div id="did-it-work-partly" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#did-it-work-partly" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Deploy, stop a machine, start it again, watch the clock. The startup now announces its decision and gets out of the way:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">15:23:34  Starting app server...
</span></span><span class="line"><span class="cl">15:23:36  release gate: this release is already prepared, starting straight away
</span></span><span class="line"><span class="cl">15:23:44  &#34;GET /healthz HTTP/1.1&#34; 200</span></span></code></pre></div></div>
<p>Ten seconds, and a second machine measured eleven. Against 25 to 33 before, two thirds of the window is gone, and the gate itself costs two seconds because it never imports Django.</p>
<p>But ten seconds is still ten seconds. A request landing in the first of them still fails. So I went looking for the rest.</p>

<h2 class="relative group">The bytecode nobody was caching
    <div id="the-bytecode-nobody-was-caching" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-bytecode-nobody-was-caching" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The image sets a line that appears in more or less every Python Dockerfile ever written:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">ENV</span> <span class="nv">PYTHONDONTWRITEBYTECODE</span><span class="o">=</span><span class="m">1</span></span></span></code></pre></div></div>
<p>It is good advice. A container should not be scribbling <code>.pyc</code> files into a layer at runtime. What I had never thought through is the other half of the bargain: if the runtime never writes bytecode, and the build never writes it either, then nothing is ever cached, and every process start compiles the entire dependency tree from source.</p>
<p>I counted:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">site-packages .py  files : 5,707
</span></span><span class="line"><span class="cl">site-packages .pyc files : 0</span></span></code></pre></div></div>
<p>Django, the REST framework, every library, recompiled on each of those 92 daily starts. Measured on a production machine:</p>
<table>
	<thead>
			<tr>
					<th><code>django.setup()</code></th>
					<th>Time</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>As shipped</td>
					<td>5.15s</td>
			</tr>
			<tr>
					<td>After <code>compileall</code></td>
					<td>3.17s</td>
			</tr>
	</tbody>
</table>
<p>The fix is one line, and <code>compileall</code> writes explicitly so it works despite the env var:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">RUN</span> python -m compileall -q /app/.venv/lib /app/myapp <span class="o">||</span> true</span></span></code></pre></div></div>
<p>Sixteen seconds of build time, once, in a cached layer.</p>
<p>While I was there I found something quieter. <code>.dockerignore</code> said:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">__pycache__/
</span></span><span class="line"><span class="cl">*.pyc</span></span></code></pre></div></div>
<p>Docker matches those against the whole relative path, not each path component, so an unanchored pattern only ever matches at the root of the build context. Every nested <code>__pycache__</code> was shipping. The image contained 218 <code>.pyc</code> files compiled by Python 3.15 on a laptop, riding along inside an image whose interpreter is 3.14 and quietly ignoring all of them. The patterns want to be <code>**/__pycache__/</code> and <code>**/*.pyc</code>.</p>
<p>Honest result: end to end this bought about a second, not two. Ten seconds became nine. The isolated benchmark oversold it, as isolated benchmarks do.</p>

<h2 class="relative group">Suspend, and the one-line setting that was blocking it
    <div id="suspend-and-the-one-line-setting-that-was-blocking-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#suspend-and-the-one-line-setting-that-was-blocking-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>At nine seconds I had run out of work to remove. What was left was one unavoidable boot: Python starting, Django importing, the WSGI app coming up on a shared core.</p>
<p>So stop booting. Fly can snapshot a machine&rsquo;s memory instead of shutting it down:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;suspend&#39;</span>   <span class="c"># was &#39;stop&#39;</span></span></span></code></pre></div></div>
<p>I went to test it and Fly refused, with the most useful error message of the whole exercise:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">failed to suspend VM: failed_precondition:
</span></span><span class="line"><span class="cl">Machines with swap cannot be suspended</span></span></code></pre></div></div>
<p>My config had this near the top, with a comment I had written myself and believed:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="c"># Swap cushion so a memory spike swaps to disk instead of getting OOM-killed.</span>
</span></span><span class="line"><span class="cl"><span class="nx">swap_size_mb</span> <span class="p">=</span> <span class="mi">2048</span></span></span></code></pre></div></div>
<p>A reasonable precaution. Was it doing anything? Every machine said:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">MemTotal:   985220 kB
</span></span><span class="line"><span class="cl">SwapTotal: 2097148 kB
</span></span><span class="line"><span class="cl">SwapFree:  2097148 kB</span></span></code></pre></div></div>
<p>Not one page had ever been swapped. And the day&rsquo;s logs held no OOM kills — the eleven lines that matched &ldquo;oom&rdquo; turned out to be requests for a JavaScript bundle whose content hash happens to contain those three letters. Meanwhile the actual memory guard was somewhere else entirely, in the app server&rsquo;s own settings, which respawn any worker past 800MB long before it can reach the 1GB ceiling.</p>
<p>So the cushion was insurance against an event that had never happened, already covered by a different mechanism, and its premium was the ability to suspend. Out it went.</p>
<table>
	<thead>
			<tr>
					<th>Wake path</th>
					<th>Time to serve</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Cold boot, this morning</td>
					<td>25 to 33s</td>
			</tr>
			<tr>
					<td>Cold boot, after the gate and bytecode</td>
					<td>9 to 10s</td>
			</tr>
			<tr>
					<td><strong>Suspended resume</strong></td>
					<td><strong>2.5 to 3.1s</strong></td>
			</tr>
	</tbody>
</table>
<p>Three trials, all within half a second. Every endpoint answered 200 in 250 to 435 milliseconds immediately after waking.</p>

<h2 class="relative group">What happens to your sockets while the machine sleeps
    <div id="what-happens-to-your-sockets-while-the-machine-sleeps" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-happens-to-your-sockets-while-the-machine-sleeps" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Nothing good, and there is no hook to fix it. Fly freezes the VM; the process is not signalled and cannot close anything on the way down. Whatever connections it held are still in its memory when it wakes, pointing at sockets the other end abandoned minutes ago.</p>
<p>You do not solve that going down. You solve it coming up, and mostly by having already made the right decisions:</p>
<ul>
<li><strong>Database connections are opened per request</strong> (<code>conn_max_age = 0</code>), so there is nothing long-lived to go stale.</li>
<li><strong>The Redis cache degrades instead of raising.</strong> A wrapper catches connection errors and returns a miss.</li>
</ul>
<p>I got to watch the second one earn its place. One resume logged exactly the failure you would expect:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">redis.exceptions.ConnectionError: Error while reading from fly-...</span></span></code></pre></div></div>
<p>and the request it happened inside returned 200 in 3.6 seconds, because a dead cache is a slow page, not an error page. Later resumes logged nothing at all.</p>
<p>If your cache client raises on a dead connection, suspend will convert every wake into a burst of 500s. Check that before you flip the setting, not after.</p>

<h2 class="relative group">The test that lied to me
    <div id="the-test-that-lied-to-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-test-that-lied-to-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My first suspend measurement said the feature was a disaster: 502, after 30 seconds, twice.</p>
<p>I had pinned the request to a specific machine with <code>fly-force-instance-id</code>, which is a lovely header for load testing one instance and a terrible one for this. Forcing an instance bypasses the proxy logic that would have woken it. Fly told me so, if I had read it as an answer rather than an error:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">machine was recently stopped and is unavailable to service request</span></span></code></pre></div></div>
<p>The number that matters comes from the path real traffic takes. Forty concurrent requests through the edge, well past the soft limit of 12, so the proxy has to scale out onto suspended machines:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">status codes:  40 × 200
</span></span><span class="line"><span class="cl">slowest:       2.1s</span></span></code></pre></div></div>
<p>Two lessons, and the second one is the expensive one. Measure the path your users take, not the path that is convenient to instrument. And when your tooling refuses to do something, read the refusal: &ldquo;Machines with swap cannot be suspended&rdquo; was the sentence that unlocked this entire afternoon.</p>

<h2 class="relative group">What I would check in your app
    <div id="what-i-would-check-in-your-app" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-would-check-in-your-app" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you run anything on a platform that stops idle instances, these questions took me an afternoon and would have saved me weeks of quiet failure:</p>
<ol>
<li><strong>How long does a cold start take?</strong> Not how long the container takes to run, how long until it answers a real request. If you cannot answer in seconds, measure it before you need to.</li>
<li><strong>What does your entrypoint do per start that belongs to a deploy?</strong> Migrations, static file collection, cache warming, index building. All fine once. All expensive ninety-two times.</li>
<li><strong>Is anything caching your bytecode?</strong> <code>PYTHONDONTWRITEBYTECODE</code> with no <code>compileall</code> in the build means the answer is no.</li>
<li><strong>Can you suspend instead of stop?</strong> And if not, what is stopping you? Mine was a swap file that had never been touched.</li>
<li><strong>Does your edge see errors your origin does not?</strong> Compare the two. A gap between them is not a reporting quirk, it is requests dying in the space between, and your application logs will never show it.</li>
</ol>
<p>The health checks were green the whole time, incidentally. They always are: a health check runs against a machine that is already up. It cannot report on the thirty seconds before it existed.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
