<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Tech Stack · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/tech-stack/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Sat, 07 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/tech-stack/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Monitoring Django on Fly.io: The DevOps Tools I Kept and Dropped</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/</guid>
      <pubDate>Thu, 22 Oct 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>DevOps</category>
      <category>Monitoring</category>
      <category>Observability</category>
      <category>Site Reliability</category>
      <category>Fly.io</category>
      <category>Django</category>
      <category>Cloudflare</category>
      <category>Sentry</category>
      <category>Crash Reporting</category>
      <category>Small Teams</category>
      <category>Tech Stack</category>
      <description>Monitoring two small Django apps on Fly.io for about US$2 a month: errors to chat, logs in S3 via DuckDB, Fly&#39;s metrics, crash reports, and what I dropped.</description>
      <content:encoded><![CDATA[<p>For two small products, a marketplace and community app I build and an estimating app for tradespeople, both on Fly.io&rsquo;s cheapest machines, my DevOps monitoring comes down to five things: ERROR logs posted to a chat channel, every log line archived in object storage and queried with DuckDB, Fly&rsquo;s built-in Prometheus metrics, phone crash reports sent through Sentry&rsquo;s open-source SDK to my own API, and one summary post each morning.</p>
<p>There&rsquo;s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here&rsquo;s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.</p>

<h2 class="relative group">What I mean by monitoring, and how small this is
    <div id="what-i-mean-by-monitoring-and-how-small-this-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-mean-by-monitoring-and-how-small-this-is" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>By DevOps monitoring I mean finding out that something&rsquo;s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.</p>
<p>Last year I wrote about <a href="https://jared.lynskey.co.nz/en/posts/2025/2025-10-13-devops-monitoring/" >setting up monitoring that developers actually use</a>, mostly with teams in mind: a cloud provider&rsquo;s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly&rsquo;s shared-cpu-1x machines, as described in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/" >My Current Stack</a>, and I&rsquo;m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly&rsquo;s metrics in September had the marketplace&rsquo;s API averaging 1% CPU. This isn&rsquo;t an enterprise stack, and I won&rsquo;t dress it up as one.</p>
<p>Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy&rsquo;s, on <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-01-cloudflare-warp-dev-network/" >my workstation</a>.</p>

<h2 class="relative group">The list
    <div id="the-list" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-list" aria-label="Anchor">#</a>
    </span>
    
</h2>
<table>
	<thead>
			<tr>
					<th>Tool</th>
					<th>What it watches</th>
					<th>Kept or dropped, and why</th>
					<th>Cost</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A team chat channel, fed by a Django logging handler</td>
					<td>ERRORs from the API, WebSocket server and worker; deploys; bursts of wrong sign-in codes</td>
					<td>Kept. It&rsquo;s the one place I look without being asked</td>
					<td>Free</td>
			</tr>
			<tr>
					<td>Fly&rsquo;s log shipper to an S3-compatible bucket, queried with DuckDB</td>
					<td>Every log line from the marketplace&rsquo;s apps</td>
					<td>Kept. Fly&rsquo;s own buffer holds minutes; this holds everything, in SQL</td>
					<td>About US$2 a month, plus storage</td>
			</tr>
			<tr>
					<td>Fly&rsquo;s built-in metrics (Prometheus API)</td>
					<td>CPU, memory, burst credit, responses; the managed Postgres exporter</td>
					<td>Kept. It showed the database stalls weren&rsquo;t my code</td>
					<td>Comes with Fly</td>
			</tr>
			<tr>
					<td>Fly health checks on <code>/api/health/</code> (both apps)</td>
					<td>Whether a machine should get traffic</td>
					<td>Kept for routing. Dropped as proof that users are fine</td>
					<td>Comes with Fly</td>
			</tr>
			<tr>
					<td>Sentry&rsquo;s SDK, its DSN pointed at my own API</td>
					<td>App crashes on phones</td>
					<td>Kept the SDK. Never signed up for sentry.io</td>
					<td>Nothing on top of the API</td>
			</tr>
			<tr>
					<td>A morning summary in the team chat</td>
					<td>Yesterday&rsquo;s errors, usage, search, chats and board</td>
					<td>Kept. Moved off a CI cron job onto my Mac</td>
					<td>Free</td>
			</tr>
			<tr>
					<td>An app analytics SDK (both apps), and a search engine&rsquo;s webmaster console</td>
					<td>Usage on web, iOS and Android; how search sees the site</td>
					<td>Kept, read through their APIs</td>
					<td>Free</td>
			</tr>
			<tr>
					<td>Cloudflare&rsquo;s GraphQL analytics</td>
					<td>Who is behind a status code or a path</td>
					<td>Kept, grouped by user agent before I believe it</td>
					<td>Free plan, one day per query</td>
			</tr>
			<tr>
					<td>A hosted over-the-air update service&rsquo;s dashboard</td>
					<td>Which over-the-air release phones run; crash rollbacks</td>
					<td>Moved to it from an admin list that couldn&rsquo;t tell me</td>
					<td>Part of the update service</td>
			</tr>
			<tr>
					<td>A small probe script</td>
					<td>Failed requests during a deploy or restart</td>
					<td>Kept. It hits the origin, so a cache can&rsquo;t hide a gap</td>
					<td>Free</td>
			</tr>
			<tr>
					<td><code>flyctl logs</code></td>
					<td>The last few minutes of an app&rsquo;s output</td>
					<td>Kept as a live tail. Dropped as the record</td>
					<td>Comes with Fly</td>
			</tr>
			<tr>
					<td>Cloudflare Early Hints</td>
					<td>Nothing; it&rsquo;s a speed feature</td>
					<td>Kept, once the origin sent a <code>Link</code> header. Its cache misses were every 504 on the zone</td>
					<td>Free</td>
			</tr>
	</tbody>
</table>
<p>Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly&rsquo;s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)</p>

<h2 class="relative group">Errors: a logging handler posting to chat
    <div id="errors-a-logging-handler-posting-to-chat" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#errors-a-logging-handler-posting-to-chat" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Until September, <code>flyctl logs</code> was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don&rsquo;t alert.</p>
<p>The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can&rsquo;t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand.</p>
<p>The channel also carries what I want to read errors against: deploys post when they start, when they&rsquo;re up and where they failed, with the commit and who ran it, and so do over-the-air updates and <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-14-web-releases-without-redeploying/" >web releases</a>. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.</p>
<p>In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process&rsquo;s start command, and writes the same line to that process&rsquo;s log, so an id I can&rsquo;t find means a process that doesn&rsquo;t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.</p>

<h2 class="relative group">Logs: Fly&rsquo;s shipper to S3, queried with DuckDB
    <div id="logs-flys-shipper-to-s3-queried-with-duckdb" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#logs-flys-shipper-to-s3-queried-with-duckdb" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly&rsquo;s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly&rsquo;s log shipper, which is Vector subscribed to the organisation&rsquo;s log stream, with one change to its S3 sink:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">sinks</span><span class="p">.</span><span class="nx">aws_s3</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s2">&#34;aws_s3&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">bucket</span> <span class="p">=</span> <span class="s2">&#34;myapp-logs&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">compression</span> <span class="p">=</span> <span class="s2">&#34;gzip&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">key_prefix</span> <span class="p">=</span> <span class="s2">&#34;logs/{{fly.app.name}}/%F/&#34;</span>   <span class="c"># one folder per app per day</span>
</span></span><span class="line"><span class="cl">  <span class="nx">batch</span><span class="p">.</span><span class="nx">timeout_secs</span> <span class="p">=</span> <span class="mi">300</span>                  <span class="c"># a quiet app still lands its lines</span></span></span></code></pre></div></div>
<p>Every line from the marketplace&rsquo;s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="n">MACRO</span><span class="w"> </span><span class="n">logs</span><span class="p">(</span><span class="n">app</span><span class="p">,</span><span class="w"> </span><span class="k">day</span><span class="w"> </span><span class="p">:</span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;*&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">TABLE</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">read_json_auto</span><span class="p">(</span><span class="s1">&#39;s3://myapp-logs/logs/&#39;</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="s1">&#39;/&#39;</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="k">day</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="s1">&#39;/*.log.gz&#39;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                               </span><span class="n">union_by_name</span><span class="w"> </span><span class="p">:</span><span class="o">=</span><span class="w"> </span><span class="k">true</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">timestamp</span><span class="p">,</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="p">(</span><span class="s1">&#39;myapp-api&#39;</span><span class="p">,</span><span class="w"> </span><span class="s1">&#39;2026-09-06&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">LIKE</span><span class="w"> </span><span class="s1">&#39;% 500 %&#39;</span><span class="p">;</span></span></span></code></pre></div></div>
<p>That macro is how I measured the cold starts in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >the scale-to-zero post</a>: the startup script announces each phase, every line carries the machine&rsquo;s id, and the rest is a <code>GROUP BY</code>.</p>
<p>So it&rsquo;s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.</p>

<h2 class="relative group">Metrics: Fly&rsquo;s built-in Prometheus
    <div id="metrics-flys-built-in-prometheus" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#metrics-flys-built-in-prometheus" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly runs a managed Prometheus for each organisation, so I don&rsquo;t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (<code>fly_instance_cpu_balance</code> matters on a shared CPU), responses and concurrency per app. Two things I wish I&rsquo;d known first. A token from <code>fly tokens create</code> goes in as <code>Authorization: FlyV1 &lt;token&gt;</code>; <code>Bearer</code> only works with the personal token from <code>flyctl auth token</code>, and a read-only token gets &ldquo;something went wrong resolving organization&rdquo;. And it keeps about two weeks (Fly says roughly 15 days).</p>
<p>It earned its place in September, when the logs had &ldquo;Could not open a database connection&rdquo; several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary&rsquo;s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres&rsquo;s start time never moved. Postgres hadn&rsquo;t restarted; the host under it was stalling. I sent Fly&rsquo;s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.</p>

<h2 class="relative group">Crash reports: Sentry&rsquo;s SDK, pointed at my own API
    <div id="crash-reports-sentrys-sdk-pointed-at-my-own-api" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#crash-reports-sentrys-sdk-pointed-at-my-own-api" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry&rsquo;s open-source SDK, but its DSN points at an endpoint in my own Django API that accepts Sentry&rsquo;s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It&rsquo;s errors only.</p>
<p>I didn&rsquo;t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven&rsquo;t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.</p>

<h2 class="relative group">The only dashboard is a message
    <div id="the-only-dashboard-is-a-message" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-only-dashboard-is-a-message" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I haven&rsquo;t built a dashboard for either app. Last year&rsquo;s post had a test for them: if nobody opens one unprompted, it isn&rsquo;t earning its place. A message that arrives on its own doesn&rsquo;t need opening, so the closest thing I have is a post in the team chat every morning.</p>
<p>It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can&rsquo;t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel.</p>
<p>Then failed health checks, yesterday&rsquo;s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what I&rsquo;ve spent. A section that can&rsquo;t load says so, and the post still goes out.</p>
<p>Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month&rsquo;s Android totals with them taken out.</p>
<p>Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an &ldquo;is active&rdquo; checkbox, and I had no idea how many phones had taken one. The hosted service&rsquo;s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-13-moving-to-codemagic-patch/" >the move to a hosted update service</a> separately.</p>

<h2 class="relative group">What I dropped, or stopped believing
    <div id="what-i-dropped-or-stopped-believing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-dropped-or-stopped-believing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>None of this was a paid tool. Mostly it was things I&rsquo;d been treating as signals that weren&rsquo;t, and one job running in the wrong place.</p>

<h3 class="relative group">Cloudflare&rsquo;s 504 rate, until I grouped it by user agent
    <div id="cloudflares-504-rate-until-i-grouped-it-by-user-agent" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cloudflares-504-rate-until-i-grouped-it-by-user-agent" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>In September the Cloudflare dashboard showed what looked like 22% of a day&rsquo;s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app&rsquo;s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >the scale-to-zero post</a> as the cause.</p>
<p>The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare&rsquo;s own Early Hints cache lookups (user agents <code>bastion early hints</code> and <code>nginx-ssl early hints</code>), which never reach the origin: a 504 on one of those lines is a cache miss, not a timeout. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn&rsquo;t doing anything for me because the origin sent no <code>Link</code> header, so I turned it off, and later <a href="https://jared.lynskey.co.nz/en/posts/2026/cloudflare-504-early-hints/" >made it work</a>.</p>
<p>The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user.</p>
<p>So this is the rule I&rsquo;d most like to pass on: before you believe a CDN&rsquo;s error rate, group it by user agent and check it isn&rsquo;t the CDN&rsquo;s own machinery. On Cloudflare&rsquo;s free plan that&rsquo;s a GraphQL query on <code>httpRequestsAdaptiveGroups</code>, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn&rsquo;t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.</p>

<h3 class="relative group">A green health check
    <div id="a-green-health-check" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-green-health-check" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Fly checks <code>/api/health/</code> on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it&rsquo;s good at. But a health check runs against a machine that&rsquo;s already up, so it can&rsquo;t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now.</p>
<p>Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain <code>flyctl deploy</code> leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.</p>
<p>The health check has also caused two outages of its own. In July, Django&rsquo;s <code>SECURE_SSL_REDIRECT</code> answered Fly&rsquo;s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an <code>except Exception</code> in the health view let a &ldquo;connection already closed&rdquo; error escape as a 500, and Fly pulled the machine in exactly the case the view&rsquo;s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.</p>
<p>The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.</p>

<h3 class="relative group">A scheduled job in CI
    <div id="a-scheduled-job-in-ci" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-scheduled-job-in-ci" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project&rsquo;s service-account key, which can do anything in that project, and I didn&rsquo;t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it&rsquo;s past eight in Auckland and today&rsquo;s post hasn&rsquo;t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.</p>

<h2 class="relative group">If I were starting again
    <div id="if-i-were-starting-again" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#if-i-were-starting-again" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I&rsquo;d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That&rsquo;s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I&rsquo;d stop checking them.</p>
<p>I&rsquo;d leave tracing until there&rsquo;s a second service for a request to cross, and a hosted APM until there&rsquo;s a team that needs shared dashboards, an on-call rota and a budget for someone else&rsquo;s bill. That&rsquo;s when the tools in last year&rsquo;s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-monitoring-tools-two-products/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Django on Fly.io: Load Testing and Autoscaling on Minimal Compute</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/</guid>
      <pubDate>Wed, 22 Jul 2026 00:00:00 +0000</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Django</category>
      <category>Python</category>
      <category>React Native</category>
      <category>Expo</category>
      <category>Fly.io</category>
      <category>PostgreSQL</category>
      <category>WebSockets</category>
      <category>Scaling</category>
      <category>Load Testing</category>
      <category>Performance</category>
      <category>Tech Stack</category>
      <description>How I run two Django apps on Fly.io&#39;s shared-cpu-1x machines: make one machine fast, load-test it to find where latency collapses, and set Fly.io autoscaling just below it.</description>
      <content:encoded><![CDATA[<p>To run Django on Fly.io cheaply, I make one small shared-cpu-1x machine as fast as I can, load-test it until latency collapses, and set Fly&rsquo;s autoscaler to start a second machine just before that point. For the Django load testing I use Locust with a step-load ramp (and, on the other app, a small Python script that sweeps concurrency), watching p95 latency and error rates. When traffic drops, Fly stops the idle machines again, so the baseline cost is one small VM per app.</p>
<p>I&rsquo;m currently shipping two products as a solo developer: a <strong>field estimation app</strong> for tradespeople who quote jobs on site, and a <strong>marketplace and community app</strong> with listings and real-time messaging.</p>
<p>Different products, different users, nearly identical stack — and both run entirely on <strong>shared-cpu-1x machines</strong>, the cheapest tier Fly.io offers. That constraint is deliberate. Instead of sizing servers for peak load, I squeeze as much as possible out of one small machine, measure exactly where it collapses, and let Fly&rsquo;s autoscaler add and remove machines around that measured limit. This post covers the optimisations that make that possible, the load-test numbers behind the thresholds, and how scaling up and down actually works on Fly.io.</p>

<h2 class="relative group">The Philosophy: Measure One Machine, Then Multiply
    <div id="the-philosophy-measure-one-machine-then-multiply" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-philosophy-measure-one-machine-then-multiply" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The scaling strategy for both apps is the same three steps:</p>
<ol>
<li>Make a single small machine as fast as it can reasonably be.</li>
<li>Load-test it until it breaks, and note the concurrency level where latency collapses.</li>
<li>Configure Fly&rsquo;s autoscaler to start a second machine just before that point, and stop idle machines when traffic drops.</li>
</ol>
<p>Capacity then scales roughly linearly with machine count, and the baseline cost is one small VM per app. Everything below serves one of those three steps.</p>

<h2 class="relative group">Making One Small Machine Fast
    <div id="making-one-small-machine-fast" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#making-one-small-machine-fast" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">Granian: WSGI for the API, uvloop ASGI for WebSockets
    <div id="granian-wsgi-for-the-api-uvloop-asgi-for-websockets" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#granian-wsgi-for-the-api-uvloop-asgi-for-websockets" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Both backends are Django 6 + Django REST Framework. The API is served by <strong>Granian</strong>, a Rust-based server — but under <strong>WSGI with a blocking thread pool</strong>, not ASGI:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">granian --interface wsgi myapp.wsgi:application <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers <span class="m">1</span> --blocking-threads <span class="m">16</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers-max-rss <span class="m">1400</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    --workers-kill-timeout <span class="m">15</span> --respawn-failed-workers</span></span></code></pre></div></div>
<p>I started on <code>--interface asgi --loop uvloop</code>, and it was the wrong call. Every HTTP view in these apps is sync, and Django&rsquo;s ASGI handler runs sync views <code>thread_sensitive</code> — one at a time per worker. So ASGI gave me roughly one concurrent request per worker <em>and</em> loaded the whole asyncio/uvloop/Channels stack into RAM for nothing. WSGI with <code>--blocking-threads</code> runs those same sync views in a real thread pool: one worker × 16 threads = 16 concurrent I/O-bound requests (DB reads, S3 presigns) at lower resident memory than the async build.</p>
<p>Why a single worker? The VM is <code>shared-cpu-1x</code> — one core. A second worker can&rsquo;t run Python in parallel on one core (GIL + one CPU), but it does cost a whole extra resident Django process; Granian warns about exactly this. Sixteen threads on one worker is the honest ceiling of a 1-vCPU box for I/O-bound work. <code>--workers-max-rss 1400</code> respawns the worker if it drifts past its memory cap, so a runaway PDF or image job gets recycled instead of walking the machine toward OOM.</p>
<p>WebSockets moved out to their own Granian process — an <strong>ASGI</strong> app with uvloop and <code>--ws</code>, one worker for long-lived connections. The sync API needs no ASGI path at all: when a view wants to push to a socket it calls <code>async_to_sync(channel_layer.group_send)</code> over Redis, so the message still crosses to the WebSocket process. Two servers, each running the interface it&rsquo;s actually good at. Local Docker runs the same WSGI thread pool, so my laptop approximates one production machine when testing.</p>

<h3 class="relative group">Rust in the Hot Paths
    <div id="rust-in-the-hot-paths" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rust-in-the-hot-paths" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Wherever a Rust-backed drop-in exists for a Python hot path, I take it:</p>
<ul>
<li><strong>orjson</strong> via <code>drf-orjson-renderer</code> — every API response serialises through Rust instead of the stdlib JSON encoder.</li>
<li><strong>blurhash-rs</strong> — image placeholders computed at upload time.</li>
<li><strong>uuid-utils</strong> — Rust UUID generation.</li>
<li><strong>psycopg 3 (binary)</strong> — C-accelerated Postgres driver.</li>
</ul>
<p>None of these change the architecture; they just raise the ceiling of what one vCPU can do before you need a second machine.</p>

<h3 class="relative group">Kill the Hidden Latency Taxes
    <div id="kill-the-hidden-latency-taxes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#kill-the-hidden-latency-taxes" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The single biggest win in either codebase wasn&rsquo;t an algorithm — it was connection reuse. Under Granian&rsquo;s thread pool, every new thread was building a fresh boto3 S3 client, and every endpoint that returns a photo or PDF paid that cost. A small custom storage class now caches the client once per process instead:</p>
<blockquote><p>Signed URLs went from <strong>~150ms to ~1ms</strong> on new threads.</p>
</blockquote><p>The same lesson applies to the database: Fly&rsquo;s Postgres sits behind PgBouncer, so the app runs <code>conn_max_age=0</code> and lets the pooler own connection reuse, with a 5-second <code>connect_timeout</code> to fail fast on stale SSL instead of hanging a worker.</p>

<h3 class="relative group">Cache Where the Bottleneck Actually Is
    <div id="cache-where-the-bottleneck-actually-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cache-where-the-bottleneck-actually-is" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Load testing the marketplace revealed something worth internalising: <strong>the ceiling was Postgres connections, not CPU</strong>. Browse traffic is overwhelmingly anonymous and identical across users, and under load those identical list queries saturate the connection pool while the CPU sits idle.</p>
<p>The fix is a Redis cache for anonymous list pages, with per-namespace version counters for invalidation — a write bumps one counter (O(1), no key bookkeeping) and every cached page for that resource is instantly stale. A short TTL backstops anything missed. Serving browse pages straight from Redis removes the real bottleneck and lets the small machine spend its connections on logged-in traffic.</p>

<h3 class="relative group">Background Work Without a Broker
    <div id="background-work-without-a-broker" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#background-work-without-a-broker" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Both apps use <strong>django-tasks-db</strong>, Django&rsquo;s database-backed task framework, instead of Celery. A separate <code>db_worker</code> process polls the database for jobs — image processing, PDF generation, machine translation, reminders. No broker, no extra service to size and monitor, and the API process never blocks on heavy work. The worker even connects straight to Postgres, bypassing PgBouncer, since long-running jobs don&rsquo;t play well with transaction pooling.</p>

<h3 class="relative group">Memory: jemalloc and a Swap Cushion
    <div id="memory-jemalloc-and-a-swap-cushion" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#memory-jemalloc-and-a-swap-cushion" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Image and PDF processing spike memory in ways steady-state sizing doesn&rsquo;t predict. Two cheap mitigations: <strong>jemalloc</strong> preloaded in the Docker image to reduce fragmentation, and <strong>swap on every machine</strong> (2GB on the API machines, 1GB on the WebSocket machine) so a spike swaps to disk instead of getting the process OOM-killed. Swap is not a performance feature — it&rsquo;s an insurance policy that lets me keep memory allocations small.</p>

<h2 class="relative group">Django Load Testing: Finding the Knee
    <div id="django-load-testing-finding-the-knee" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#django-load-testing-finding-the-knee" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Thresholds are only defensible if they&rsquo;re measured. Each project has its own harness:</p>
<ul>
<li>The estimation app has a small Python stress script that sweeps concurrency levels and reports RPS, p50/p95/p99 and error rates. It runs against a few hundred generated sample records, so the list endpoints are queried at realistic data volumes.</li>
<li>The marketplace uses <strong>Locust</strong> with a step-load shape that ramps 50 → 500 users in 30-second steps to find the saturation point, with endpoint weights matching real browse-heavy traffic, plus a second, smaller suite that ramps 5 → 50 virtual users against production with a p95 &lt; 2s threshold.</li>
</ul>
<p>Representative numbers from the estimation app&rsquo;s sweep (one Granian worker × 16 blocking threads, Docker on my dev machine):</p>
<table>
	<thead>
			<tr>
					<th>Endpoint</th>
					<th>Concurrency</th>
					<th>RPS</th>
					<th>p95</th>
					<th>Errors</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>health</td>
					<td>25</td>
					<td>602</td>
					<td>52ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>heavy list</td>
					<td>25</td>
					<td>309</td>
					<td>260ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>light list</td>
					<td>25</td>
					<td>357</td>
					<td>61ms</td>
					<td>0%</td>
			</tr>
			<tr>
					<td>heavy list</td>
					<td>50</td>
					<td>309</td>
					<td>347ms</td>
					<td>0%</td>
			</tr>
	</tbody>
</table>
<p>And the number that drives everything else comes from load-testing the marketplace on a <strong>single shared-cpu-1x machine</strong>. On the old ASGI build this was a hard wall: sync views serialised to roughly <strong>three usable slots</strong>, so p95 held under 300ms up to ~5 concurrent requests and then collapsed past <strong>~8–10</strong> as everything queued behind the thread-sensitive lock. Moving the API to a WSGI thread pool is what pushed that knee out — <strong>16 real concurrent slots</strong> per machine instead of ~3 — and that new ceiling is the whole basis of the autoscaling config.</p>

<h2 class="relative group">How Scaling Works on Fly.io
    <div id="how-scaling-works-on-flyio" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-scaling-works-on-flyio" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Fly&rsquo;s model is simple: your app is a fleet of identical machines behind a proxy, and the proxy counts load per machine. You declare two numbers:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">concurrency</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s1">&#39;requests&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">soft_limit</span> <span class="p">=</span> <span class="mi">12</span>   <span class="c"># past this, start another machine</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hard_limit</span> <span class="p">=</span> <span class="mi">24</span>   <span class="c"># past this, shed load instead of queueing</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;stop&#39;</span>
</span></span><span class="line"><span class="cl"><span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl"><span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p><strong>Scaling up:</strong> when in-flight requests on a machine exceed the soft limit, the proxy starts a stopped machine and routes new traffic there. The WSGI thread pool gives each machine 16 real concurrent slots, so the soft limit sits at 12 — about 75% of the pool, so machine #2 starts while threads still have headroom rather than after users are seeing multi-second latency. The hard limit of 24 (1.5× the pool) is the circuit breaker: short bursts queue on threads, and beyond it the proxy sheds load rather than letting one machine melt down. (The old ASGI build ran 8/20, sized for its ~3 real slots — widening the pool is what let both numbers move up.)</p>
<p><strong>Scaling down:</strong> when traffic drops, Fly stops idle machines automatically, down to <code>min_machines_running = 1</code>. A stopped machine costs nothing for compute, only its rootfs storage, so the baseline cost of each app is literally one small VM with burst capacity always ready.</p>
<p>I originally wrote here that a stopped machine &ldquo;cold-starts in well under a second.&rdquo; That was wrong, and expensively so. When I finally measured it instead of assuming it, a cold start took <strong>25 to 33 seconds</strong>, and a request that lands on a machine still booting has to wait out all of it. I first blamed those cold starts for a 22% failure rate on my CDN dashboard, while every health check stayed green. That number turned out to be the CDN&rsquo;s own Early Hints probes failing against the origin; no real client got a 504. The cold starts were real all the same.</p>
<p>The fix is not in the scaling numbers above; those are fine. It is in what the machine does on the way up, and in asking Fly to suspend machines rather than stop them, which turns a wake into a ~3 second memory restore. I wrote that up separately: <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/" >Scale to Zero, and Every Restart Is a 30-Second Outage</a> covers the diagnosis and what to change. If you are running anything on Fly with <code>auto_stop_machines</code>, read that one before you trust this paragraph.</p>
<p>One subtlety: the concurrency type is <code>requests</code>, not <code>connections</code>. Modern clients hold keep-alive connections open, so counting connections barely moves under real load and the autoscaler never triggers. Counting in-flight requests tracks actual work.</p>

<h3 class="relative group">WebSockets Scale on a Different Axis
    <div id="websockets-scale-on-a-different-axis" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#websockets-scale-on-a-different-axis" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The marketplace splits its WebSocket server into a <strong>second Fly app</strong> sharing the same Docker image. The API app runs one worker × 16 blocking threads under WSGI and counts <em>requests</em> (soft limit 12). The WebSocket app runs 1 worker under ASGI with uvloop on a <strong>512MB</strong> machine and counts <em>connections</em>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">.</span><span class="nx">concurrency</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">type</span> <span class="p">=</span> <span class="s1">&#39;connections&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">soft_limit</span> <span class="p">=</span> <span class="mi">1500</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hard_limit</span> <span class="p">=</span> <span class="mi">2000</span></span></span></code></pre></div></div>
<p>A long-lived socket is nearly free to hold open but occupies a slot for hours, while an HTTP request occupies a slot for milliseconds — the same autoscaling signal can&rsquo;t serve both. Splitting them means each fleet scales on its own honest metric, and idle WebSocket capacity costs 512MB, not 2GB. On the Channels side, the Redis layer runs with <code>capacity: 1500</code> and 60-second message expiry, plus socket keepalives every 30 seconds because Fly&rsquo;s Redis proxy idle-closes the long-lived BRPOP sockets that Channels depends on.</p>

<h2 class="relative group">Offloading Compute to the Edge and the Client
    <div id="offloading-compute-to-the-edge-and-the-client" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#offloading-compute-to-the-edge-and-the-client" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The cheapest server work is the work the server never does:</p>
<ul>
<li><strong>Static and media files</strong> live in Fly&rsquo;s S3-compatible object storage, served through a CDN with signed URLs for private media — zero image bytes transit the Django machines.</li>
<li><strong>AI inference runs on the user&rsquo;s phone.</strong> Both apps embed a small local LLM (a quantised 4B model, around 2.7GB) that runs on the device. Zero inference cost server-side, and it works offline.</li>
<li><strong>Offline-first mobile</strong> (SQLite on the device and a sync queue with retries) means the apps batch writes and tolerate dead zones instead of hammering the API with chatty per-keystroke requests.</li>
</ul>

<h2 class="relative group">The Rest of the Stack, Briefly
    <div id="the-rest-of-the-stack-briefly" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-rest-of-the-stack-briefly" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Auth is a hosted identity provider plus WebAuthn passkeys, and WebSocket connections are authenticated before the upgrade. Postgres in production, SQLite in development, and a couple of off-the-shelf Django packages for the admin and audit history. Mobile is React Native and TypeScript on Expo, with Expo Router and Tailwind, over-the-air updates from a self-hosted Expo Updates server, and a subscriptions service for in-app purchases. Quality gates: pytest and mypy on the backend, unit and end-to-end tests on the client.</p>

<h2 class="relative group">What I&rsquo;d Tell You to Steal
    <div id="what-id-tell-you-to-steal" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-tell-you-to-steal" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The transferable idea isn&rsquo;t any single library — it&rsquo;s the loop. Make one cheap machine genuinely fast (Rust hot paths, connection reuse, cache in front of the real bottleneck, offload what you can to CDN and client). Load-test until you find the exact concurrency where it breaks. Set the autoscaler&rsquo;s soft limit at that knee, let idle machines stop, and keep the minimum fleet at one. Two production apps, real-time features, AI assistants — and the steady-state bill is a couple of the cheapest VMs on the platform.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-07-22-my-current-stack/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
