<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Performance · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/performance/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Sat, 21 Nov 2026 17:52:23 +0900</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/performance/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>A Closed Pull Request, a Fork and a PyPI Package: Fast HTML to Markdown for Python</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-10-18-pyhtml2text/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-10-18-pyhtml2text/</guid>
      <pubDate>Sun, 18 Oct 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Python</category>
      <category>Rust</category>
      <category>PyPI</category>
      <category>Open Source</category>
      <category>Markdown</category>
      <category>LLM</category>
      <category>PyO3</category>
      <category>Performance</category>
      <description>I needed relative links resolved when converting HTML to Markdown for AI agents. The upstream Rust crate couldn&#39;t take my pull request because it was written with an LLM, so I forked it and published pyhtml2text: Python bindings with base_url, a new html2markdown, profile-guided wheels, and a patched parser that handles deeply nested HTML in linear time. Here&#39;s why it exists, what&#39;s in it, and how I checked the output.</description>
      <content:encoded><![CDATA[<p>On 15 October I published <a href="https://pypi.org/project/pyhtml2text/"  target="_blank" rel="noreferrer">pyhtml2text</a> on PyPI. It converts HTML to plain text or Markdown and is built on <a href="https://github.com/jugglerchris/rust-html2text"  target="_blank" rel="noreferrer">rust-html2text</a>, Chris Emerson&rsquo;s Rust crate. Converting to Markdown, it was between 3 and 21 times faster than the usual Python converters on the pages I measured. It can also resolve relative links against the page&rsquo;s URL, so the links in the output still work once they&rsquo;re away from the page.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">pip install pyhtml2text</span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pyhtml2text</span> <span class="kn">import</span> <span class="n">html2markdown</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">html</span> <span class="o">=</span> <span class="s2">&#34;&#34;&#34;&lt;h2&gt;Before you start&lt;/h2&gt;
</span></span></span><span class="line"><span class="cl"><span class="s2">&lt;p&gt;Read the &lt;a href=&#34;setup.html&#34;&gt;setup guide&lt;/a&gt; and &lt;b&gt;back up&lt;/b&gt; your data.&lt;/p&gt;
</span></span></span><span class="line"><span class="cl"><span class="s2">&lt;img src=&#34;https://jared.lynskey.co.nz/img/diagram.png&#34; alt=&#34;How it fits together&#34;&gt;
</span></span></span><span class="line"><span class="cl"><span class="s2">&lt;table&gt;&lt;tr&gt;&lt;th&gt;Plan&lt;/th&gt;&lt;th&gt;Price&lt;/th&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Basic&lt;/td&gt;&lt;td&gt;Free&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">html2markdown</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">base_url</span><span class="o">=</span><span class="s2">&#34;https://example.com/docs/intro.html&#34;</span><span class="p">))</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-markdown" data-lang="markdown"><span class="line"><span class="cl"><span class="gu">## Before you start
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Read the [<span class="nt">setup guide</span>](<span class="na">https://example.com/docs/setup.html</span>) and <span class="gs">**back up**</span> your data.
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">![<span class="nt">How it fits together</span>](<span class="na">https://example.com/img/diagram.png</span>)
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">| Plan | Price |
</span></span><span class="line"><span class="cl">| --- | --- |
</span></span><span class="line"><span class="cl">| Basic | Free |</span></span></code></pre></div></div>
<p>It started as a pull request to rust-html2text. The maintainer closed it because it was written with an LLM, so pyhtml2text runs on a fork instead. This post covers how that happened, what&rsquo;s in the fork and the package, and how I checked that the Markdown is right.</p>

<h2 class="relative group">Why I needed it
    <div id="why-i-needed-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-i-needed-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A marketplace and community app I build answers <code>Accept: text/markdown</code> with Markdown instead of HTML, for AI agents and LLM clients that would rather not wade through nav bars and footers. I covered the edge side of that in the <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-cloudflare-crawlers-cache/" >post on crawlers and the cache</a>. Pages that come from app data build their Markdown directly, because their HTML is an empty JavaScript shell. Everything else is static HTML (the marketing, legal and blog pages), and that gets converted.</p>
<p>The problem with converting it is the links. A static site is full of links like <code>/en/posts/...</code> and <code>../setup/</code>. They&rsquo;re fine in a browser, which knows the page they came from. An agent reading the Markdown has no such context, so it can&rsquo;t follow <code>/login</code>. It needs <code>https://example.com/login</code>.</p>
<p>The pure-Python <a href="https://pypi.org/project/html2text/"  target="_blank" rel="noreferrer">html2text</a> has a <code>baseurl</code> option for this, and that&rsquo;s what I used at first. But it&rsquo;s slow, and its Markdown has problems I&rsquo;ll get to below. rust-html2text was fast and well tested, but it had no way to resolve links.</p>

<h2 class="relative group">Asking upstream first
    <div id="asking-upstream-first" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#asking-upstream-first" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On 29 August I opened <a href="https://github.com/jugglerchris/rust-html2text/issues/287"  target="_blank" rel="noreferrer">issue #287</a> with the use case and the behaviour I wanted: RFC 3986 resolution, with absolute URLs and <code>mailto:</code>, <code>data:</code> and <code>#fragment</code> links left alone. A few minutes later I opened <a href="https://github.com/jugglerchris/rust-html2text/pull/288"  target="_blank" rel="noreferrer">pull request #288</a>. It added a <code>Config::base_url(...)</code> option, and I said in the issue that it was a starting point and that I&rsquo;d understand if the maintainer wanted to take a different approach.</p>
<p>On 11 October the maintainer closed the PR:</p>
<blockquote><p>Thanks for the contribution, but I&rsquo;m unable to accept this as it appears to be LLM generated. (I have now added a brief policy to the README.md - it wasn&rsquo;t previously stated)</p>
</blockquote><p>and replied on the issue:</p>
<blockquote><p>Hi - this is a reasonable feature request, thanks! I have closed the PR as it appears to be LLM-written which I can&rsquo;t accept, but thanks for the attempt.</p>
<p>A short HTML sample and expected output would definitely help get the feature implemented!</p>
</blockquote><p>The README now has this policy:</p>
<blockquote><p>At this time, this project is not able to accept code contributions produced with generative AI/LLMs due to the uncertain copyright status.</p>
</blockquote><p>That was a correct read: I wrote that code with an AI coding assistant. I don&rsquo;t think the policy is unreasonable, either. The copyright status of LLM output isn&rsquo;t settled, and it&rsquo;s the maintainer who would carry the risk of merging it into an MIT crate that other projects depend on. It&rsquo;s their project, and the reply was polite. The feature request is still open.</p>
<p>I still needed the feature, though, and arguing with a maintainer over their own contribution policy isn&rsquo;t a good way to get it. So I kept the change on a fork and built the Python package on top of that, and both say clearly that they aren&rsquo;t upstream.</p>

<h2 class="relative group">What&rsquo;s in the fork
    <div id="whats-in-the-fork" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-in-the-fork" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The fork is the <code>pyhtml2text-perf</code> branch of my rust-html2text fork. I&rsquo;ve kept it to three small commits so that it rebases cleanly onto upstream. The main one is the <code>Config::base_url</code> option from the PR: relative links are resolved per <a href="https://www.rfc-editor.org/rfc/rfc3986#section-5.3"  target="_blank" rel="noreferrer">RFC 3986 §5.3</a>, and without a base URL the output is unchanged. Another stops the crate rebuilding its style rules on every call, which took a short snippet from 7.3 µs to 5.6 µs. That&rsquo;s nothing on a big page, but it adds up when you call it a few thousand times. The third just exposes what the Markdown converter below needs from the crate.</p>

<h2 class="relative group">The Python package
    <div id="the-python-package" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-python-package" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><a href="https://github.com/jaredlynskey/pyhtml2text"  target="_blank" rel="noreferrer">pyhtml2text</a> is built with PyO3 and maturin, and it has two functions:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pyhtml2text</span> <span class="kn">import</span> <span class="n">html2text</span><span class="p">,</span> <span class="n">html2markdown</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">html2text</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">width</span><span class="o">=</span><span class="mi">80</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">base_url</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">decorate</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">link_footnotes</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="n">table_borders</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">wrap_links</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">max_wrap_width</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">raw</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">html2markdown</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">base_url</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span></span></span></code></pre></div></div>
<p><code>html2text</code> is rust-html2text&rsquo;s plain-text renderer, which is built for terminals and the <code>text/plain</code> part of an email:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">html2text</span><span class="p">(</span><span class="s1">&#39;&lt;p&gt;Please &lt;a href=&#34;https://jared.lynskey.co.nz/login&#34;&gt;sign in&lt;/a&gt; first.&lt;/p&gt;&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="n">base_url</span><span class="o">=</span><span class="s2">&#34;https://example.com/docs/page.html&#34;</span><span class="p">))</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Please [sign in][1] first.
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">[1]: https://example.com/login</span></span></code></pre></div></div>
<p>Some things about the bindings that may not be obvious:</p>
<ul>
<li><strong>Both functions take <code>str</code> or UTF-8 <code>bytes</code></strong>, so you can pass <code>response.content</code> without decoding it first. The input is borrowed, not copied.</li>
<li><strong>The GIL is released while converting.</strong> It&rsquo;s pure Rust from start to finish, so threads scale across cores. The module also says it doesn&rsquo;t need the GIL, so free-threaded Python 3.14t doesn&rsquo;t turn it back on when the module is imported.</li>
<li><strong>A panic comes back as <code>ValueError</code>.</strong> By default PyO3 raises a <code>PanicException</code>, which <code>except Exception</code> doesn&rsquo;t catch. A web worker that thinks it has handled conversion errors would crash on that, so the bindings catch it and re-raise it as a <code>ValueError</code>.</li>
<li><strong>There are 30 wheels:</strong> abi3 wheels for CPython 3.10 and later, plus free-threaded 3.14t wheels, across Linux (glibc and musl), macOS and Windows on every architecture maturin&rsquo;s CI can build. So nobody needs a Rust toolchain to install it.</li>
</ul>

<h2 class="relative group">Making it fast
    <div id="making-it-fast" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#making-it-fast" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Version 0.2.0 was 1.5 to 1.7 times faster than 0.1.0 on real pages, and twice as fast on short snippets, with byte-for-byte identical output. Most of that came from profile-guided optimisation: CI builds an instrumented wheel, runs it over a training set of documentation pages and RFCs, and uses the profile to build the real wheel. That&rsquo;s about 25% faster on pages outside the training set, on the platforms where CI can run the instrumented build natively. The rest came from switching to the mimalloc allocator, which copes far better with lots of small allocations, and from the style-rule change in the fork.</p>
<p>To check the speed-ups didn&rsquo;t change anything, I hashed the output for a corpus of real pages under every set of options, plus edge cases like empty input and very deep nesting. Each change had to leave every hash the same.</p>
<p>Here are the median times on an Apple Silicon Mac, converting to Markdown:</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Encyclopaedia article (1 MB)</th>
					<th>News site front page (370 KB)</th>
					<th>Python docs page (245 KB)</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://pypi.org/project/markdownify/"  target="_blank" rel="noreferrer">markdownify</a></td>
					<td>184 ms</td>
					<td>28 ms</td>
					<td>97 ms</td>
			</tr>
			<tr>
					<td><a href="https://pypi.org/project/html2text/"  target="_blank" rel="noreferrer">html2text</a> (pure Python)</td>
					<td>73 ms</td>
					<td>13 ms</td>
					<td>37 ms</td>
			</tr>
			<tr>
					<td><a href="https://pypi.org/project/html-to-markdown/"  target="_blank" rel="noreferrer">html-to-markdown</a></td>
					<td>44 ms</td>
					<td>5.6 ms</td>
					<td>15 ms</td>
			</tr>
			<tr>
					<td><strong>pyhtml2text</strong></td>
					<td><strong>14 ms</strong></td>
					<td><strong>1.7 ms</strong></td>
					<td><strong>4.5 ms</strong></td>
			</tr>
	</tbody>
</table>
<p>And converting to plain text:</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Encyclopaedia article (780 KB)</th>
					<th>News site front page (370 KB)</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://pypi.org/project/html2text/"  target="_blank" rel="noreferrer">html2text</a> (pure Python)</td>
					<td>71.5 ms</td>
					<td>14.9 ms</td>
			</tr>
			<tr>
					<td><a href="https://pypi.org/project/html2text-rs/"  target="_blank" rel="noreferrer">html2text-rs</a></td>
					<td>15.2 ms</td>
					<td>2.5 ms</td>
			</tr>
			<tr>
					<td><strong>pyhtml2text</strong></td>
					<td><strong>10.9 ms</strong></td>
					<td><strong>1.6 ms</strong></td>
			</tr>
	</tbody>
</table>

<h2 class="relative group">Deeply nested HTML, and a patch to html5ever
    <div id="deeply-nested-html-and-a-patch-to-html5ever" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#deeply-nested-html-and-a-patch-to-html5ever" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Nobody writes 40,000 nested lists by hand. But if you convert pages you didn&rsquo;t write, sooner or later something hostile or broken will send you one. With the stock parser, 40,000 nested <code>&lt;ul&gt;&lt;li&gt;</code> took <strong>6 seconds</strong> to parse, and that&rsquo;s a worker stuck on a single request.</p>
<p>The cause was in <a href="https://github.com/servo/html5ever"  target="_blank" rel="noreferrer">html5ever</a>, the HTML5 parser rust-html2text uses. Most block-level start tags have to check whether there&rsquo;s an open <code>&lt;p&gt;</code> element &ldquo;in button scope&rdquo;, and html5ever answered that by walking the whole stack of open elements. The deeper the nesting, the longer each tag took, so the total time grew with the square of the depth.</p>
<p>The package ships html5ever 0.39.0 with one patch that keeps the answer up to date as elements are opened and closed, so the check no longer depends on depth. The same document now parses in <strong>0.14 seconds</strong>. To be sure the patch changed nothing else, I compared the DOMs from the stock and patched parsers on every html5lib tree-construction test, about 20 real pages and 300,000 random tag-soup documents, and they were all identical.</p>
<p>The Markdown converter has its own limits for the same reason. No input can overflow the thread&rsquo;s stack, blockquotes and lists stop indenting past 32 levels so the output can&rsquo;t grow quadratically, and a table whose expanded grid would pass 100,000 cells is written as its cells alone.</p>

<h2 class="relative group">html2markdown
    <div id="html2markdown" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#html2markdown" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Version 0.3.0 added <code>html2markdown</code>, and most of the work in the package went into it. rust-html2text&rsquo;s renderer makes text that looks like Markdown, but it&rsquo;s wrapped for a terminal and isn&rsquo;t meant to be parsed. So <code>html2markdown</code> only borrows the parser and writes its own <a href="https://commonmark.org/"  target="_blank" rel="noreferrer">CommonMark</a>, plus GitHub-style pipe tables and <code>~~strikethrough~~</code>:</p>
<ul>
<li><code>#</code> headings, <code>-</code> bullets, <code>_emphasis_</code> and <code>**strong**</code>.</li>
<li>Inline links and images, made absolute when <code>base_url</code> is given. A link whose only content is an icon uses its <code>aria-label</code> or <code>title</code> as its text.</li>
<li>Fenced code blocks, tagged with the language when the page names one (<code>class=&quot;language-python&quot;</code>).</li>
<li>Pipe tables. Cells that span rows or columns become empty cells, a table without a header row gets an empty one, and tables used for page layout are written as paragraphs.</li>
<li>No line wrapping. Hidden content (<code>hidden</code>, <code>aria-hidden</code>, <code>display: none</code>), scripts, styles and form controls are left out.</li>
</ul>

<h3 class="relative group">Escaping only what needs it
    <div id="escaping-only-what-needs-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#escaping-only-what-needs-it" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Some HTML-to-Markdown converters escape every character that could be Markdown syntax. markdownify, for example, escapes every <code>_</code> and <code>*</code> by default. The output is correct, but it&rsquo;s full of <code>snake\_case</code> and <code>5 \* 3</code>, which is harder for people to read and a waste of tokens for a model. pyhtml2text escapes a character only where a CommonMark parser would otherwise read it as syntax:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">html2markdown</span><span class="p">(</span><span class="s1">&#39;&lt;p&gt;Use &lt;code&gt;snake_case&lt;/code&gt;, 5 * 3, $5 and &#39;</span>
</span></span><span class="line"><span class="cl">              <span class="s1">&#39;&lt;a href=&#34;../setup.html&#34;&gt;setup&lt;/a&gt;&lt;/p&gt;&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">              <span class="n">base_url</span><span class="o">=</span><span class="s2">&#34;https://example.com/docs/intro.html&#34;</span><span class="p">)</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-markdown" data-lang="markdown"><span class="line"><span class="cl">Use <span class="sb">`snake_case`</span>, 5 * 3, $5 and [<span class="nt">setup</span>](<span class="na">https://example.com/setup.html</span>)</span></span></code></pre></div></div>

<h3 class="relative group">Bold before a Korean particle
    <div id="bold-before-a-korean-particle" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#bold-before-a-korean-particle" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>This one came from my app&rsquo;s own pages. In Korean the grammatical particle goes straight after the noun with no space, so a bold term with brackets in it looks like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-html" data-lang="html"><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;&lt;</span><span class="nt">b</span><span class="p">&gt;</span>타이타닉(Titanic)<span class="p">&lt;/</span><span class="nt">b</span><span class="p">&gt;</span>의 감독<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span></span></span></code></pre></div></div>
<p>The obvious Markdown is <code>**타이타닉(Titanic)**의</code>, which is what markdownify writes, and it doesn&rsquo;t work. CommonMark&rsquo;s flanking rules don&rsquo;t let <code>**</code> close after punctuation unless a space or more punctuation follows it, and <code>의</code> is a letter, so the asterisks stay as literal text. The pure-Python html2text avoids this by adding a space:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-markdown" data-lang="markdown"><span class="line"><span class="cl">**타이타닉(Titanic)** 의 감독</span></span></code></pre></div></div>
<p>That renders as bold, but it changes the Korean. The particle is now a separate word, which is a spelling mistake. pyhtml2text moves the punctuation outside the markers instead:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-markdown" data-lang="markdown"><span class="line"><span class="cl">**타이타닉(Titanic**)의 감독</span></span></code></pre></div></div>
<p>The bracket is no longer bold, but the emphasis still parses and every character of the original is kept. The same applies to Japanese and Chinese, and to any language that doesn&rsquo;t put a space before what follows.</p>

<h3 class="relative group">Checking the output by reading it back
    <div id="checking-the-output-by-reading-it-back" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#checking-the-output-by-reading-it-back" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>I don&rsquo;t trust a converter because its unit tests pass, because the bugs in Markdown output are in combinations nobody thought to write a test for. So the output gets checked by reading it back.</p>
<p>While converting, each paragraph is parsed again as CommonMark to confirm the emphasis reads as intended. If it doesn&rsquo;t, the paragraph is written again with the other emphasis style (<code>*</code> instead of <code>_</code>, or the other way round), and if neither works the text, links and code are kept without the formatting, so no words are lost. The test suite does a round trip through a different Markdown parser and compares every visible word, link target and image source with the original HTML. Text eaten by syntax that wasn&rsquo;t escaped, a link that didn&rsquo;t parse or a stray <code>*</code> all show up as differences. It runs over real pages and tens of thousands of random documents, and shrinks any failure down to a minimal case.</p>

<h2 class="relative group">In production
    <div id="in-production" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#in-production" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My app&rsquo;s Markdown negotiation switched from the pure-Python html2text to pyhtml2text the day 0.3.0 came out. On one of its guide pages the conversion went from 4.2 ms to 0.36 ms, about 12 times faster. The links are absolute, the tables are real pipe tables, and the Korean bold keeps its particles attached. The abi3 wheels already cover Python 3.14 on x86_64 and arm64, so the container image needed nothing new to install it.</p>

<h2 class="relative group">Credits, and where to report bugs
    <div id="credits-and-where-to-report-bugs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#credits-and-where-to-report-bugs" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The plain-text rendering is rust-html2text, by Chris Emerson, under the MIT licence. pyhtml2text adds a fork of it, a patched copy of html5ever (MIT/Apache-2.0) and the Markdown converter. The package isn&rsquo;t affiliated with or endorsed by either project, so <strong>please report problems to <a href="https://github.com/jaredlynskey/pyhtml2text/issues"  target="_blank" rel="noreferrer">pyhtml2text&rsquo;s issue tracker</a></strong>, not upstream. Any bug in this code is mine to fix, and the upstream maintainers shouldn&rsquo;t get reports about code they declined to take.</p>
<p>If you&rsquo;re converting HTML in Python, especially for a model to read, give it a try:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">pip install pyhtml2text</span></span></code></pre></div></div>
<p>The source is on <a href="https://github.com/jaredlynskey/pyhtml2text"  target="_blank" rel="noreferrer">GitHub</a>, and the changelog lists everything in 0.1.0 to 0.3.0.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-10-18-pyhtml2text/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Fly.io Cold Starts with Scale to Zero: From 33 Seconds to 3</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/</guid>
      <pubDate>Wed, 09 Sep 2026 00:00:00 +1200</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Fly.io</category>
      <category>Django</category>
      <category>DevOps</category>
      <category>Cloudflare</category>
      <category>Performance</category>
      <category>Cold Starts</category>
      <category>Redis</category>
      <category>Debugging</category>
      <description>Why my Fly.io machines took 25 to 33 seconds to cold start with auto_stop_machines, and how a release lock, compiled bytecode and suspend got wake-ups down to about 3 seconds.</description>
      <content:encoded><![CDATA[<p><strong>Update, 22 October 2026:</strong> the headline number in this post is wrong. On the day I published it I grouped the zone&rsquo;s 504s by user agent, and all 7,344 of them between 5 and 9 September came from Cloudflare&rsquo;s own Early Hints probes (<code>bastion early hints</code> and <code>nginx-ssl early hints</code>). Those get no response from the Fly origin, are logged as 504s, and mirror real request paths, so they look exactly like user failures. No real client (the app, browsers, curl, search crawlers) got a 504 in that window. I turned Early Hints off the same day: the origin sends no preload headers, so it was doing nothing useful. The cold starts below were real, and so are the fixes, but the 22% was never what my users saw. Before you treat a Cloudflare 5xx rate as an outage, check which user agents are getting them.</p>
<p><strong>Update, 4 November 2026:</strong> one more correction. Those 504s aren&rsquo;t probes the origin fails to answer. They&rsquo;re Cloudflare&rsquo;s own Early Hints cache lookups, which never reach the origin at all, and a 504 there only means nothing was cached for that URL. Turning Early Hints off wasn&rsquo;t the best fix either: sending a <code>Link</code> response header made the feature work and turned those misses into hits. The full story is in <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >Cloudflare 504 errors that aren&rsquo;t real: Early Hints and HTTP 103</a>.</p>
<p>With <code>auto_stop_machines</code> on, any request that lands on a stopped Fly.io machine waits for a full boot, and mine took 25 to 33 seconds because the entrypoint ran Django migrations and <code>collectstatic</code> on every machine start, and nothing had ever cached the bytecode. Running that release work once per image (a Redis lock keyed on <code>FLY_IMAGE_REF</code>) and adding <code>compileall</code> to the build got a cold boot down to about nine seconds. Switching <code>auto_stop_machines</code> from <code>'stop'</code> to <code>'suspend'</code>, which meant dropping a swap file Fly won&rsquo;t suspend with, got wake-ups to about three.</p>
<p>I went looking for a traffic spike on my Cloudflare dashboard and found something much worse underneath it: 22% of every request my app served in a day had failed. No alert had fired. Every health check was green. The logs looked perfectly healthy, because the failing requests never reached the application at all.</p>
<p>The cause turned out to be my own container startup script, doing sensible work at exactly the wrong time.</p>

<h2 class="relative group">The spike was a red herring
    <div id="the-spike-was-a-red-herring" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-spike-was-a-red-herring" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The thing I set out to explain was one hour that took about 4,900 requests against a median of 121. Forty-two unique visitors. That is 110 requests per visitor, which is not how humans browse.</p>
<p>It was a scanner. A single bot walking through invented subdomains, asking each one for secrets:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">api-stage-asia.example.com/.env.local
</span></span><span class="line"><span class="cl">api-us-east-1-demo.example.com/.env
</span></span><span class="line"><span class="cl">api-eu-uat.example.com/wp-config.php~
</span></span><span class="line"><span class="cl">app-test-us-east-1.example.com/actuator/configprops
</span></span><span class="line"><span class="cl">prod-api-use2.example.com/.env.local
</span></span><span class="line"><span class="cl">relay.example.com/.env.production</span></span></code></pre></div></div>
<p>Thirty or so hostnames, every <code>.env</code> variant you can think of, plus a Java framework&rsquo;s actuator endpoints on a Django app. All of them got a 301 to my canonical host and nothing was served. Boring, and genuinely harmless.</p>
<p>But while I had the analytics open, I grouped the same day by status code. That is where the real problem was.</p>
<table>
	<thead>
			<tr>
					<th>Status</th>
					<th>Requests in 24h</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Total</td>
					<td>11,350</td>
			</tr>
			<tr>
					<td>504 Gateway Timeout</td>
					<td>2,474</td>
			</tr>
	</tbody>
</table>
<p>Twenty-two percent. And not spread evenly across the site, but concentrated exactly where it hurts:</p>
<table>
	<thead>
			<tr>
					<th>Endpoint</th>
					<th>504s</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>notifications unread count</td>
					<td>621</td>
			</tr>
			<tr>
					<td>notifications list</td>
					<td>614</td>
			</tr>
			<tr>
					<td>messages unread count</td>
					<td>469</td>
			</tr>
			<tr>
					<td>feed cursor</td>
					<td>210</td>
			</tr>
	</tbody>
</table>
<p>Those are the endpoints my mobile app polls every 30 seconds. So this was not an abstract error-rate number. It was every user&rsquo;s phone, quietly failing to load their unread badge, all day, for weeks probably.</p>

<h2 class="relative group">Why the logs looked fine
    <div id="why-the-logs-looked-fine" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-the-logs-looked-fine" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My first instinct was that the app had become slow. It had not. I pulled the application&rsquo;s own request lines for one of the worst hours out of the log archive:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[2026-09-08 01:00:29] &#34;GET /api/notifications/unread/&#34; 304 61.615
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:29] &#34;GET /api/notifications/&#34;        304 63.568
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:59] &#34;GET /api/notifications/unread/&#34; 304 58.742
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:59] &#34;GET /api/notifications/&#34;        304 105.438</span></span></code></pre></div></div>
<p>Forty-four to a hundred milliseconds, 304s all the way down. The origin was fast.</p>
<p>That is the important clue, and it is worth stating as a rule: <strong>if your edge reports errors your origin has never heard of, the requests are dying before they arrive.</strong> Stop reading application logs and start looking at what sits in front of them.</p>
<p>In my case that is Fly.io&rsquo;s proxy, and the answer was in how my machines are scheduled.</p>

<h2 class="relative group">Scale to zero is a promise about money, not latency
    <div id="scale-to-zero-is-a-promise-about-money-not-latency" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#scale-to-zero-is-a-promise-about-money-not-latency" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My config looks like this, and on the face of it it is reasonable:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;stop&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">  <span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p>Idle machines stop. Traffic starts them again. You pay for what you use. For a small app with bursty traffic this is exactly what you want, and it is why I had ten machines defined but almost always one running.</p>
<p>The bill it hides is latency. When a request arrives for a stopped machine, somebody has to wait for that machine to boot. If booting takes longer than the proxy&rsquo;s patience, that somebody gets a 504.</p>
<p>So: how long did my machines take to boot? I had never measured. The log archive knows, because the startup script announces each phase, and every line carries the machine id:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">WITH</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">SELECT</span><span class="w"> </span><span class="n">instance</span><span class="p">,</span><span class="w"> </span><span class="k">timestamp</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">         </span><span class="k">CASE</span><span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Starting app server%&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;begin&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Migrations complete%&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;migrated&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">LIKE</span><span class="w">  </span><span class="s1">&#39;%&#34;GET /healthz%&#39;</span><span class="w">       </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;serving&#39;</span><span class="w"> </span><span class="k">END</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">phase</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">WHERE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;myapp&#39;</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="k">timestamp</span><span class="p">::</span><span class="nb">date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;2026-09-08&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="n">instance</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;begin&#39;</span><span class="p">)</span><span class="w">    </span><span class="k">AS</span><span class="w"> </span><span class="n">started</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;migrated&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">migrated</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;serving&#39;</span><span class="p">)</span><span class="w">  </span><span class="k">AS</span><span class="w"> </span><span class="n">first_serve</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">instance</span></span></span></code></pre></div></div>
<p>The answer, consistently, across every machine:</p>
<table>
	<thead>
			<tr>
					<th>Phase</th>
					<th>Duration</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Boot to migrations finished</td>
					<td>10 to 13 seconds</td>
			</tr>
			<tr>
					<td>Migrations to first served request</td>
					<td>8 to 19 seconds</td>
			</tr>
			<tr>
					<td><strong>Total</strong></td>
					<td><strong>25 to 33 seconds</strong></td>
			</tr>
	</tbody>
</table>
<p>Then I counted how often that happened:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">starts</span><span class="p">,</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="k">DISTINCT</span><span class="w"> </span><span class="n">instance</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">machines</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">WHERE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;myapp&#39;</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="k">timestamp</span><span class="p">::</span><span class="nb">date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;2026-09-08&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">AND</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Starting app server%&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">-- starts: 92, machines: 10</span></span></span></code></pre></div></div>
<p>Ninety-two cold starts in one day. Each one a half-minute window in which anything routed to that machine timed out. Twenty-two percent stopped looking mysterious.</p>

<h2 class="relative group">The cause was in my own entrypoint
    <div id="the-cause-was-in-my-own-entrypoint" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-cause-was-in-my-own-entrypoint" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Here is what every single machine ran before it would answer a request:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># entrypoint.sh, run on every machine start</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Running migrations...&#34;</span>
</span></span><span class="line"><span class="cl">python manage.py migrate --noinput
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Running collectstatic in background...&#34;</span>
</span></span><span class="line"><span class="cl">python manage.py collectstatic --noinput <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">exec</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span>    <span class="c1"># then start the app server</span></span></span></code></pre></div></div>
<p>Read that again with &ldquo;92 times a day&rdquo; in mind.</p>
<p><code>migrate</code> is a full Django boot: import every model, connect to Postgres, query the migrations table, decide there is nothing to do. On a shared CPU that is ten seconds to accomplish nothing. <code>collectstatic</code> is a second full Django boot that then uploads static files to object storage, competing for the same single core that the web server is trying to start on. The app server is the third boot.</p>
<p>Three Python interpreters, two of them doing work that had already been done, every time a machine woke up.</p>
<p>The reason it was written that way is good, and it is probably why yours looks similar: migrations have to run somewhere, Fly&rsquo;s <code>release_command</code> had hung on me before, and the entrypoint is the one place guaranteed to execute on deploy. It is a correct place to put deploy work. It is just not a correct place to put it <em>per machine start</em>, and once your machines stop and start on their own, those are no longer the same event.</p>
<p>That is the whole bug, and I think it is a common one:</p>
<blockquote><p>Work that belongs to a <strong>release</strong> was being run once per <strong>machine start</strong>. Scale-to-zero turned one event into ninety-two.</p>
</blockquote>
<h2 class="relative group">The fix: claim the release once
    <div id="the-fix-claim-the-release-once" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-fix-claim-the-release-once" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Migrations do not need to run on the eighth machine to boot a given image. They need to run once for that image, and everyone else needs to know it has happened.</p>
<p>That is a lock with a name that changes per deploy. Fly hands you the name for free in <code>FLY_IMAGE_REF</code>, and I already had Redis. The whole thing is a small script that runs before Django exists, so the common answer costs milliseconds rather than a framework boot:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># release_gate.py: runs before Django exists. Exit code 0 means &#34;prepare this release&#34;.</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">PREPARE</span><span class="p">,</span> <span class="n">SKIP</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl"><span class="n">LOCK_TTL</span><span class="p">,</span> <span class="n">WAIT</span><span class="p">,</span> <span class="n">POLL</span> <span class="o">=</span> <span class="mi">300</span><span class="p">,</span> <span class="mi">240</span><span class="p">,</span> <span class="mi">2</span>
</span></span><span class="line"><span class="cl"><span class="n">_sleep</span><span class="p">,</span> <span class="n">_monotonic</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span>    <span class="c1"># aliases, so tests can replace them</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">release_id</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Something that changes exactly when the deployed image does.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_IMAGE_REF&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_MACHINE_VERSION&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">claim</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">release</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">release</span><span class="p">:</span>   <span class="c1"># no Redis, or no idea which release: behave as before</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">PREPARE</span>
</span></span><span class="line"><span class="cl">    <span class="n">done_key</span><span class="p">,</span> <span class="n">lock_key</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;release:</span><span class="si">{</span><span class="n">release</span><span class="si">}</span><span class="s2">:done&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;release:</span><span class="si">{</span><span class="n">release</span><span class="si">}</span><span class="s2">:lock&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">done_key</span><span class="p">):</span>            <span class="c1"># somebody already did it for this image</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">SKIP</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">owner</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_MACHINE_ID&#34;</span><span class="p">,</span> <span class="s2">&#34;unknown&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="n">LOCK_TTL</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">PREPARE</span>                  <span class="c1"># we won the race; we do the work</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Someone else is preparing. Wait for them, because serving requests</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># against a half-migrated schema is worse than starting slowly.</span>
</span></span><span class="line"><span class="cl">    <span class="n">deadline</span> <span class="o">=</span> <span class="n">_monotonic</span><span class="p">()</span> <span class="o">+</span> <span class="n">WAIT</span>
</span></span><span class="line"><span class="cl">    <span class="k">while</span> <span class="n">_monotonic</span><span class="p">()</span> <span class="o">&lt;</span> <span class="n">deadline</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">_sleep</span><span class="p">(</span><span class="n">POLL</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">done_key</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">SKIP</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="n">LOCK_TTL</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">PREPARE</span>              <span class="c1"># the owner died and its lock expired; take over</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">PREPARE</span>                      <span class="c1"># waited long enough; do it ourselves</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># ... plus `done` (sets done_key) and a main() that treats any Redis error as PREPARE</span></span></span></code></pre></div></div>
<p>The entrypoint becomes an <code>if</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="k">if</span> python3 release_gate.py claim<span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    python manage.py migrate --noinput
</span></span><span class="line"><span class="cl">    python manage.py collectstatic --noinput <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">    python3 release_gate.py <span class="k">done</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">exec</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span></span></span></code></pre></div></div>

<h2 class="relative group">Every failure has to end in &ldquo;prepare it anyway&rdquo;
    <div id="every-failure-has-to-end-in-prepare-it-anyway" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#every-failure-has-to-end-in-prepare-it-anyway" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is the part worth copying, more than the code.</p>
<p>A gate like this sits between your users and a schema migration. If it is wrong in the direction of <em>skipping</em>, you serve requests against a database that has not been migrated, which is a genuinely bad outage. If it is wrong in the direction of <em>preparing</em>, you run an idempotent no-op migration twice and waste ten seconds.</p>
<p>Those outcomes are not symmetrical, so the code must not treat them as such. Every uncertain path returns <code>PREPARE</code>:</p>
<ul>
<li><strong>No Redis, or Redis unreachable.</strong> Prepare. This is exactly the old behaviour, so a cache outage degrades to &ldquo;slow&rdquo;, not &ldquo;broken&rdquo;.</li>
<li><strong>No image reference in the environment.</strong> Prepare. I do not know what release this is, so I cannot know whether it is ready.</li>
<li><strong>The lock&rsquo;s owner died mid-migration.</strong> Its lock expires, the next waiter takes it over and prepares.</li>
<li><strong>Waited longer than the timeout.</strong> Prepare anyway. A machine that never serves is worse than a duplicated migration.</li>
</ul>
<p>There is one deliberate asymmetry in the other direction: a machine that finds another machine actively holding the lock <em>waits</em> rather than serving. Starting slowly during a deploy is fine. Answering queries against a half-applied schema is not.</p>
<p>Watch out for one trap if you build this on a cache wrapper that swallows connection errors, as I first did. A wrapper like that returns <code>False</code> from <code>add()</code> when Redis is unreachable, which reads identically to &ldquo;someone else holds the lock&rdquo; and would have made every machine skip its migrations. That is precisely the dangerous direction. Distinguish the two by reading the lock back: a lock nobody holds means the cache is broken, not that you lost a race.</p>

<h2 class="relative group">Testing it without a Redis
    <div id="testing-it-without-a-redis" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#testing-it-without-a-redis" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The logic is worth unit testing, because the interesting paths are the ones you cannot reproduce by hand. A fake with two methods covers it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FakeRedis</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Enough of a Redis client for the gate: get, and set with nx/ex.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">nx</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">True</span></span></span></code></pre></div></div>
<p>Then drive the awkward cases by making time pass on your terms. This is why the gate calls module-level <code>_sleep</code> and <code>_monotonic</code> aliases rather than <code>time.sleep</code> directly, so a test can replace them without patching the <code>time</code> module for every other thread in the process:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_a_waiter_takes_over_when_the_owner_disappears</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">redis</span> <span class="o">=</span> <span class="n">FakeRedis</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">release_gate</span><span class="o">.</span><span class="n">claim</span><span class="p">(</span><span class="n">redis</span><span class="p">,</span> <span class="s2">&#34;img-1&#34;</span><span class="p">),</span> <span class="n">release_gate</span><span class="o">.</span><span class="n">PREPARE</span><span class="p">)</span>   <span class="c1"># the owner takes the lock</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">owner_dies</span><span class="p">(</span><span class="n">_seconds</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">redis</span><span class="o">.</span><span class="n">store</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s2">&#34;release:img-1:lock&#34;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>                              <span class="c1"># its lock expired</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">patch</span><span class="o">.</span><span class="n">object</span><span class="p">(</span><span class="n">release_gate</span><span class="p">,</span> <span class="s2">&#34;_sleep&#34;</span><span class="p">,</span> <span class="n">owner_dies</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">release_gate</span><span class="o">.</span><span class="n">claim</span><span class="p">(</span><span class="n">redis</span><span class="p">,</span> <span class="s2">&#34;img-1&#34;</span><span class="p">),</span> <span class="n">release_gate</span><span class="o">.</span><span class="n">PREPARE</span><span class="p">)</span></span></span></code></pre></div></div>

<h2 class="relative group">Did it work? Partly
    <div id="did-it-work-partly" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#did-it-work-partly" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Deploy, stop a machine, start it again, watch the clock. The startup now announces its decision and gets out of the way:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">15:23:34  Starting app server...
</span></span><span class="line"><span class="cl">15:23:36  release gate: this release is already prepared, starting straight away
</span></span><span class="line"><span class="cl">15:23:44  &#34;GET /healthz HTTP/1.1&#34; 200</span></span></code></pre></div></div>
<p>Ten seconds, and a second machine measured eleven. Against 25 to 33 before, two thirds of the window is gone, and the gate itself costs two seconds because it never imports Django.</p>
<p>But ten seconds is still ten seconds. A request landing in the first of them still fails. So I went looking for the rest.</p>

<h2 class="relative group">The bytecode nobody was caching
    <div id="the-bytecode-nobody-was-caching" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-bytecode-nobody-was-caching" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The image sets a line that appears in more or less every Python Dockerfile ever written:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">ENV</span> <span class="nv">PYTHONDONTWRITEBYTECODE</span><span class="o">=</span><span class="m">1</span></span></span></code></pre></div></div>
<p>It is good advice. A container should not be scribbling <code>.pyc</code> files into a layer at runtime. What I had never thought through is the other half of the bargain: if the runtime never writes bytecode, and the build never writes it either, then nothing is ever cached, and every process start compiles the entire dependency tree from source.</p>
<p>I counted:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">site-packages .py  files : 5,707
</span></span><span class="line"><span class="cl">site-packages .pyc files : 0</span></span></code></pre></div></div>
<p>Django, the REST framework, every library, recompiled on each of those 92 daily starts. Measured on a production machine:</p>
<table>
	<thead>
			<tr>
					<th><code>django.setup()</code></th>
					<th>Time</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>As shipped</td>
					<td>5.15s</td>
			</tr>
			<tr>
					<td>After <code>compileall</code></td>
					<td>3.17s</td>
			</tr>
	</tbody>
</table>
<p>The fix is one line, and <code>compileall</code> writes explicitly so it works despite the env var:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">RUN</span> python -m compileall -q /app/.venv/lib /app/myapp <span class="o">||</span> true</span></span></code></pre></div></div>
<p>Sixteen seconds of build time, once, in a cached layer.</p>
<p>While I was there I found something quieter. <code>.dockerignore</code> said:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">__pycache__/
</span></span><span class="line"><span class="cl">*.pyc</span></span></code></pre></div></div>
<p>Docker matches those against the whole relative path, not each path component, so an unanchored pattern only ever matches at the root of the build context. Every nested <code>__pycache__</code> was shipping. The image contained 218 <code>.pyc</code> files compiled by Python 3.15 on a laptop, riding along inside an image whose interpreter is 3.14 and quietly ignoring all of them. The patterns want to be <code>**/__pycache__/</code> and <code>**/*.pyc</code>.</p>
<p>Honest result: end to end this bought about a second, not two. Ten seconds became nine. The isolated benchmark oversold it, as isolated benchmarks do.</p>

<h2 class="relative group">Suspend, and the one-line setting that was blocking it
    <div id="suspend-and-the-one-line-setting-that-was-blocking-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#suspend-and-the-one-line-setting-that-was-blocking-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>At nine seconds I had run out of work to remove. What was left was one unavoidable boot: Python starting, Django importing, the WSGI app coming up on a shared core.</p>
<p>So stop booting. Fly can snapshot a machine&rsquo;s memory instead of shutting it down:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;suspend&#39;</span>   <span class="c"># was &#39;stop&#39;</span></span></span></code></pre></div></div>
<p>I went to test it and Fly refused, with the most useful error message of the whole exercise:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">failed to suspend VM: failed_precondition:
</span></span><span class="line"><span class="cl">Machines with swap cannot be suspended</span></span></code></pre></div></div>
<p>My config had this near the top, with a comment I had written myself and believed:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="c"># Swap cushion so a memory spike swaps to disk instead of getting OOM-killed.</span>
</span></span><span class="line"><span class="cl"><span class="nx">swap_size_mb</span> <span class="p">=</span> <span class="mi">2048</span></span></span></code></pre></div></div>
<p>A reasonable precaution. Was it doing anything? Every machine said:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">MemTotal:   985220 kB
</span></span><span class="line"><span class="cl">SwapTotal: 2097148 kB
</span></span><span class="line"><span class="cl">SwapFree:  2097148 kB</span></span></code></pre></div></div>
<p>Not one page had ever been swapped. And the day&rsquo;s logs held no OOM kills — the eleven lines that matched &ldquo;oom&rdquo; turned out to be requests for a JavaScript bundle whose content hash happens to contain those three letters. Meanwhile the actual memory guard was somewhere else entirely, in the app server&rsquo;s own settings, which respawn any worker past 800MB long before it can reach the 1GB ceiling.</p>
<p>So the cushion was insurance against an event that had never happened, already covered by a different mechanism, and its premium was the ability to suspend. Out it went.</p>
<table>
	<thead>
			<tr>
					<th>Wake path</th>
					<th>Time to serve</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Cold boot, this morning</td>
					<td>25 to 33s</td>
			</tr>
			<tr>
					<td>Cold boot, after the gate and bytecode</td>
					<td>9 to 10s</td>
			</tr>
			<tr>
					<td><strong>Suspended resume</strong></td>
					<td><strong>2.5 to 3.1s</strong></td>
			</tr>
	</tbody>
</table>
<p>Three trials, all within half a second. Every endpoint answered 200 in 250 to 435 milliseconds immediately after waking.</p>

<h2 class="relative group">What happens to your sockets while the machine sleeps
    <div id="what-happens-to-your-sockets-while-the-machine-sleeps" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-happens-to-your-sockets-while-the-machine-sleeps" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Nothing good, and there is no hook to fix it. Fly freezes the VM; the process is not signalled and cannot close anything on the way down. Whatever connections it held are still in its memory when it wakes, pointing at sockets the other end abandoned minutes ago.</p>
<p>You do not solve that going down. You solve it coming up, and mostly by having already made the right decisions:</p>
<ul>
<li><strong>Database connections are opened per request</strong> (<code>conn_max_age = 0</code>), so there is nothing long-lived to go stale.</li>
<li><strong>The Redis cache degrades instead of raising.</strong> A wrapper catches connection errors and returns a miss.</li>
</ul>
<p>I got to watch the second one earn its place. One resume logged exactly the failure you would expect:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">redis.exceptions.ConnectionError: Error while reading from fly-...</span></span></code></pre></div></div>
<p>and the request it happened inside returned 200 in 3.6 seconds, because a dead cache is a slow page, not an error page. Later resumes logged nothing at all.</p>
<p>If your cache client raises on a dead connection, suspend will convert every wake into a burst of 500s. Check that before you flip the setting, not after.</p>

<h2 class="relative group">The test that lied to me
    <div id="the-test-that-lied-to-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-test-that-lied-to-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My first suspend measurement said the feature was a disaster: 502, after 30 seconds, twice.</p>
<p>I had pinned the request to a specific machine with <code>fly-force-instance-id</code>, which is a lovely header for load testing one instance and a terrible one for this. Forcing an instance bypasses the proxy logic that would have woken it. Fly told me so, if I had read it as an answer rather than an error:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">machine was recently stopped and is unavailable to service request</span></span></code></pre></div></div>
<p>The number that matters comes from the path real traffic takes. Forty concurrent requests through the edge, well past the soft limit of 12, so the proxy has to scale out onto suspended machines:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">status codes:  40 × 200
</span></span><span class="line"><span class="cl">slowest:       2.1s</span></span></code></pre></div></div>
<p>Two lessons, and the second one is the expensive one. Measure the path your users take, not the path that is convenient to instrument. And when your tooling refuses to do something, read the refusal: &ldquo;Machines with swap cannot be suspended&rdquo; was the sentence that unlocked this entire afternoon.</p>

<h2 class="relative group">What I would check in your app
    <div id="what-i-would-check-in-your-app" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-would-check-in-your-app" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you run anything on a platform that stops idle instances, these questions took me an afternoon and would have saved me weeks of quiet failure:</p>
<ol>
<li><strong>How long does a cold start take?</strong> Not how long the container takes to run, how long until it answers a real request. If you cannot answer in seconds, measure it before you need to.</li>
<li><strong>What does your entrypoint do per start that belongs to a deploy?</strong> Migrations, static file collection, cache warming, index building. All fine once. All expensive ninety-two times.</li>
<li><strong>Is anything caching your bytecode?</strong> <code>PYTHONDONTWRITEBYTECODE</code> with no <code>compileall</code> in the build means the answer is no.</li>
<li><strong>Can you suspend instead of stop?</strong> And if not, what is stopping you? Mine was a swap file that had never been touched.</li>
<li><strong>Does your edge see errors your origin does not?</strong> Compare the two. A gap between them is not a reporting quirk, it is requests dying in the space between, and your application logs will never show it.</li>
</ol>
<p>The health checks were green the whole time, incidentally. They always are: a health check runs against a machine that is already up. It cannot report on the thirty seconds before it existed.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Django on Fly.io: Load Testing and Autoscaling on Minimal Compute</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/</guid>
      <pubDate>Wed, 22 Jul 2026 00:00:00 +0000</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Django</category>
      <category>Python</category>
      <category>React Native</category>
      <category>Expo</category>
      <category>Fly.io</category>
      <category>PostgreSQL</category>
      <category>WebSockets</category>
      <category>Scaling</category>
      <category>Load Testing</category>
      <category>Performance</category>
      <category>Tech Stack</category>
      <description>How I run two Django apps on Fly.io&#39;s shared-cpu-1x machines: make one machine fast, load-test it to find where latency collapses, and set Fly.io autoscaling just below it.</description>
      <content:encoded><![CDATA[<p>To run Django on Fly.io cheaply, I make one small shared-cpu-1x machine as fast as I can, load-test it until latency collapses, and set Fly&rsquo;s autoscaler to start a second machine just before that point. For the Django load testing I use Locust with a step-load ramp (and, on the other app, a small Python script that sweeps concurrency), watching p95 latency and error rates. When traffic drops, Fly stops the idle machines again, so the baseline cost is one small VM per app.</p>
<p>I&rsquo;m currently shipping two products as a solo developer: a <strong>field estimation app</strong> for tradespeople who quote jobs on site, and a <strong>marketplace and community app</strong> with listings and real-time messaging.</p>
<p>Different products, different users, nearly identical stack — and both run entirely on <strong>shared-cpu-1x machines</strong>, the cheapest tier Fly.io offers. That constraint is deliberate. Instead of sizing servers for peak load, I squeeze as much as possible out of one small machine, measure exactly where it collapses, and let Fly&rsquo;s autoscaler add and remove machines around that measured limit. This post covers the optimisations that make that possible, the load-test numbers behind the thresholds, and how scaling up and down actually works on Fly.io.</p>

<h2 class="relative group">The Philosophy: Measure One Machine, Then Multiply
    <div id="the-philosophy-measure-one-machine-then-multiply" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-philosophy-measure-one-machine-then-multiply" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The scaling strategy for both apps is the same three steps:</p>
<ol>
<li>Make a single small machine as fast as it can reasonably be.</li>
<li>Load-test it until it breaks, and note the concurrency level where latency collapses.</li>
<li>Configure Fly&rsquo;s autoscaler to start a second machine just before that point, and stop idle machines when traffic drops.</li>
</ol>
<p>Capacity then scales roughly linearly with machine count, and the baseline cost is one small VM per app. Everything below serves one of those three steps.</p>

<h2 class="relative group">Making One Small Machine Fast
    <div id="making-one-small-machine-fast" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#making-one-small-machine-fast" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">Granian: WSGI for the API, uvloop ASGI for WebSockets
    <div id="granian-wsgi-for-the-api-uvloop-asgi-for-websockets" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#granian-wsgi-for-the-api-uvloop-asgi-for-websockets" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Both backends are Django 6 + Django REST Framework. The API is served by <strong>Granian</strong>, a Rust-based server — but under <strong>WSGI with a blocking thread pool</strong>, not ASGI:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">granian --interface wsgi myapp.wsgi:application <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers <span class="m">1</span> --blocking-threads <span class="m">16</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers-max-rss <span class="m">1400</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers-kill-timeout <span class="m">15</span> --respawn-failed-workers</span></span></code></pre></div></div>
<p>I started on <code>--interface asgi --loop uvloop</code>, and it was the wrong call. Every HTTP view in these apps is sync, and Django&rsquo;s ASGI handler runs sync views <code>thread_sensitive</code> — one at a time per worker. So ASGI gave me roughly one concurrent request per worker <em>and</em> loaded the whole asyncio/uvloop/Channels stack into RAM for nothing. WSGI with <code>--blocking-threads</code> runs those same sync views in a real thread pool: one worker × 16 threads = 16 concurrent I/O-bound requests (DB reads, S3 presigns) at lower resident memory than the async build.</p>
<p>Why a single worker? The VM is <code>shared-cpu-1x</code> — one core. A second worker can&rsquo;t run Python in parallel on one core (GIL + one CPU), but it does cost a whole extra resident Django process; Granian warns about exactly this. Sixteen threads on one worker is the honest ceiling of a 1-vCPU box for I/O-bound work. <code>--workers-max-rss 1400</code> respawns the worker if it drifts past its memory cap, so a runaway PDF or image job gets recycled instead of walking the machine toward OOM.</p>
<p>WebSockets moved out to their own Granian process — an <strong>ASGI</strong> app with uvloop and <code>--ws</code>, one worker for long-lived connections. The sync API needs no ASGI path at all: when a view wants to push to a socket it calls <code>async_to_sync(channel_layer.group_send)</code> over Redis, so the message still crosses to the WebSocket process. Two servers, each running the interface it&rsquo;s actually good at. Local Docker runs the same WSGI thread pool, so my laptop approximates one production machine when testing.</p>

<h3 class="relative group">Rust in the Hot Paths
    <div id="rust-in-the-hot-paths" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rust-in-the-hot-paths" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Wherever a Rust-backed drop-in exists for a Python hot path, I take it:</p>
<ul>
<li><strong>orjson</strong> via <code>drf-orjson-renderer</code> — every API response serialises through Rust instead of the stdlib JSON encoder.</li>
<li><strong>blurhash-rs</strong> — image placeholders computed at upload time.</li>
<li><strong>uuid-utils</strong> — Rust UUID generation.</li>
<li><strong>psycopg 3 (binary)</strong> — C-accelerated Postgres driver.</li>
</ul>
<p>None of these change the architecture; they just raise the ceiling of what one vCPU can do before you need a second machine.</p>

<h3 class="relative group">Kill the Hidden Latency Taxes
    <div id="kill-the-hidden-latency-taxes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#kill-the-hidden-latency-taxes" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The single biggest win in either codebase wasn&rsquo;t an algorithm — it was connection reuse. Under Granian&rsquo;s thread pool, every new thread was building a fresh boto3 S3 client, and every endpoint that returns a photo or PDF paid that cost. A small custom storage class now caches the client once per process instead:</p>
<blockquote><p>Signed URLs went from <strong>~150ms to ~1ms</strong> on new threads.</p>
</blockquote><p>The same lesson applies to the database: Fly&rsquo;s Postgres sits behind PgBouncer, so the app runs <code>conn_max_age=0</code> and lets the pooler own connection reuse, with a 5-second <code>connect_timeout</code> to fail fast on stale SSL instead of hanging a worker.</p>

<h3 class="relative group">Cache Where the Bottleneck Actually Is
    <div id="cache-where-the-bottleneck-actually-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cache-where-the-bottleneck-actually-is" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Load testing the marketplace revealed something worth internalising: <strong>the ceiling was Postgres connections, not CPU</strong>. Browse traffic is overwhelmingly anonymous and identical across users, and under load those identical list queries saturate the connection pool while the CPU sits idle.</p>
<p>The fix is a Redis cache for anonymous list pages, with per-namespace version counters for invalidation — a write bumps one counter (O(1), no key bookkeeping) and every cached page for that resource is instantly stale. A short TTL backstops anything missed. Serving browse pages straight from Redis removes the real bottleneck and lets the small machine spend its connections on logged-in traffic.</p>

<h3 class="relative group">Background Work Without a Broker
    <div id="background-work-without-a-broker" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#background-work-without-a-broker" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Both apps use <strong>django-tasks-db</strong>, Django&rsquo;s database-backed task framework, instead of Celery. A separate <code>db_worker</code> process polls the database for jobs — image processing, PDF generation, machine translation, reminders. No broker, no extra service to size and monitor, and the API process never blocks on heavy work. The worker even connects straight to Postgres, bypassing PgBouncer, since long-running jobs don&rsquo;t play well with transaction pooling.</p>

<h3 class="relative group">Memory: jemalloc and a Swap Cushion
    <div id="memory-jemalloc-and-a-swap-cushion" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#memory-jemalloc-and-a-swap-cushion" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Image and PDF processing spike memory in ways steady-state sizing doesn&rsquo;t predict. Two cheap mitigations: <strong>jemalloc</strong> preloaded in the Docker image to reduce fragmentation, and <strong>swap on every machine</strong> (2GB on the API machines, 1GB on the WebSocket machine) so a spike swaps to disk instead of getting the process OOM-killed. Swap is not a performance feature — it&rsquo;s an insurance policy that lets me keep memory allocations small.</p>

<h2 class="relative group">Django Load Testing: Finding the Knee
    <div id="django-load-testing-finding-the-knee" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#django-load-testing-finding-the-knee" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Thresholds are only defensible if they&rsquo;re measured. Each project has its own harness:</p>
<ul>
<li>The estimation app has a small Python stress script that sweeps concurrency levels and reports RPS, p50/p95/p99 and error rates. It runs against a few hundred generated sample records, so the list endpoints are queried at realistic data volumes.</li>
<li>The marketplace uses <strong>Locust</strong> with a step-load shape that ramps 50 → 500 users in 30-second steps to find the saturation point, with endpoint weights matching real browse-heavy traffic, plus a second, smaller suite that ramps 5 → 50 virtual users against production with a p95 &lt; 2s threshold.</li>
</ul>
<p>Representative numbers from the estimation app&rsquo;s sweep (one Granian worker × 16 blocking threads, Docker on my dev machine):</p>
<table>
	<thead>
			<tr>
					<th>Endpoint</th>
					<th>Concurrency</th>
					<th>RPS</th>
					<th>p95</th>
					<th>Errors</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>health</td>
					<td>25</td>
					<td>602</td>
					<td>52ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>heavy list</td>
					<td>25</td>
					<td>309</td>
					<td>260ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>light list</td>
					<td>25</td>
					<td>357</td>
					<td>61ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>heavy list</td>
					<td>50</td>
					<td>309</td>
					<td>347ms</td>
					<td>0%</td>
			</tr>
	</tbody>
</table>
<p>And the number that drives everything else comes from load-testing the marketplace on a <strong>single shared-cpu-1x machine</strong>. On the old ASGI build this was a hard wall: sync views serialised to roughly <strong>three usable slots</strong>, so p95 held under 300ms up to ~5 concurrent requests and then collapsed past <strong>~8–10</strong> as everything queued behind the thread-sensitive lock. Moving the API to a WSGI thread pool is what pushed that knee out — <strong>16 real concurrent slots</strong> per machine instead of ~3 — and that new ceiling is the whole basis of the autoscaling config.</p>

<h2 class="relative group">How Scaling Works on Fly.io
    <div id="how-scaling-works-on-flyio" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-scaling-works-on-flyio" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly&rsquo;s model is simple: your app is a fleet of identical machines behind a proxy, and the proxy counts load per machine. You declare two numbers:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">concurrency</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s1">&#39;requests&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">soft_limit</span> <span class="p">=</span> <span class="mi">12</span>   <span class="c"># past this, start another machine</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hard_limit</span> <span class="p">=</span> <span class="mi">24</span>   <span class="c"># past this, shed load instead of queueing</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;stop&#39;</span>
</span></span><span class="line"><span class="cl"><span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl"><span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p><strong>Scaling up:</strong> when in-flight requests on a machine exceed the soft limit, the proxy starts a stopped machine and routes new traffic there. The WSGI thread pool gives each machine 16 real concurrent slots, so the soft limit sits at 12 — about 75% of the pool, so machine #2 starts while threads still have headroom rather than after users are seeing multi-second latency. The hard limit of 24 (1.5× the pool) is the circuit breaker: short bursts queue on threads, and beyond it the proxy sheds load rather than letting one machine melt down. (The old ASGI build ran 8/20, sized for its ~3 real slots — widening the pool is what let both numbers move up.)</p>
<p><strong>Scaling down:</strong> when traffic drops, Fly stops idle machines automatically, down to <code>min_machines_running = 1</code>. A stopped machine costs nothing for compute, only its rootfs storage, so the baseline cost of each app is literally one small VM with burst capacity always ready.</p>
<p>I originally wrote here that a stopped machine &ldquo;cold-starts in well under a second.&rdquo; That was wrong, and expensively so. When I finally measured it instead of assuming it, a cold start took <strong>25 to 33 seconds</strong>, and a request that lands on a machine still booting has to wait out all of it. I first blamed those cold starts for a 22% failure rate on my CDN dashboard, while every health check stayed green. That number turned out to be the CDN&rsquo;s own Early Hints probes failing against the origin; no real client got a 504. The cold starts were real all the same.</p>
<p>The fix is not in the scaling numbers above; those are fine. It is in what the machine does on the way up, and in asking Fly to suspend machines rather than stop them, which turns a wake into a ~3 second memory restore. I wrote that up separately: <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >Scale to Zero, and Every Restart Is a 30-Second Outage</a> covers the diagnosis and what to change. If you are running anything on Fly with <code>auto_stop_machines</code>, read that one before you trust this paragraph.</p>
<p>One subtlety: the concurrency type is <code>requests</code>, not <code>connections</code>. Modern clients hold keep-alive connections open, so counting connections barely moves under real load and the autoscaler never triggers. Counting in-flight requests tracks actual work.</p>

<h3 class="relative group">WebSockets Scale on a Different Axis
    <div id="websockets-scale-on-a-different-axis" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#websockets-scale-on-a-different-axis" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The marketplace splits its WebSocket server into a <strong>second Fly app</strong> sharing the same Docker image. The API app runs one worker × 16 blocking threads under WSGI and counts <em>requests</em> (soft limit 12). The WebSocket app runs 1 worker under ASGI with uvloop on a <strong>512MB</strong> machine and counts <em>connections</em>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">concurrency</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s1">&#39;connections&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">soft_limit</span> <span class="p">=</span> <span class="mi">1500</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hard_limit</span> <span class="p">=</span> <span class="mi">2000</span></span></span></code></pre></div></div>
<p>A long-lived socket is nearly free to hold open but occupies a slot for hours, while an HTTP request occupies a slot for milliseconds — the same autoscaling signal can&rsquo;t serve both. Splitting them means each fleet scales on its own honest metric, and idle WebSocket capacity costs 512MB, not 2GB. On the Channels side, the Redis layer runs with <code>capacity: 1500</code> and 60-second message expiry, plus socket keepalives every 30 seconds because Fly&rsquo;s Redis proxy idle-closes the long-lived BRPOP sockets that Channels depends on.</p>

<h2 class="relative group">Offloading Compute to the Edge and the Client
    <div id="offloading-compute-to-the-edge-and-the-client" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#offloading-compute-to-the-edge-and-the-client" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The cheapest server work is the work the server never does:</p>
<ul>
<li><strong>Static and media files</strong> live in Fly&rsquo;s S3-compatible object storage, served through a CDN with signed URLs for private media — zero image bytes transit the Django machines.</li>
<li><strong>AI inference runs on the user&rsquo;s phone.</strong> Both apps embed a small local LLM (a quantised 4B model, around 2.7GB) that runs on the device. Zero inference cost server-side, and it works offline.</li>
<li><strong>Offline-first mobile</strong> (SQLite on the device and a sync queue with retries) means the apps batch writes and tolerate dead zones instead of hammering the API with chatty per-keystroke requests.</li>
</ul>

<h2 class="relative group">The Rest of the Stack, Briefly
    <div id="the-rest-of-the-stack-briefly" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-rest-of-the-stack-briefly" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Auth is a hosted identity provider plus WebAuthn passkeys, and WebSocket connections are authenticated before the upgrade. Postgres in production, SQLite in development, and a couple of off-the-shelf Django packages for the admin and audit history. Mobile is React Native and TypeScript on Expo, with Expo Router and Tailwind, over-the-air updates from a self-hosted Expo Updates server, and a subscriptions service for in-app purchases. Quality gates: pytest and mypy on the backend, unit and end-to-end tests on the client.</p>

<h2 class="relative group">What I&rsquo;d Tell You to Steal
    <div id="what-id-tell-you-to-steal" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-tell-you-to-steal" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The transferable idea isn&rsquo;t any single library — it&rsquo;s the loop. Make one cheap machine genuinely fast (Rust hot paths, connection reuse, cache in front of the real bottleneck, offload what you can to CDN and client). Load-test until you find the exact concurrency where it breaks. Set the autoscaler&rsquo;s soft limit at that knee, let idle machines stop, and keep the minimum fleet at one. Two production apps, real-time features, AI assistants — and the steady-state bill is a couple of the cheapest VMs on the platform.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
