<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Testing · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/testing/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Fri, 13 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/testing/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>How to Set Up AWS Emulator, the Free LocalStack Alternative</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/</guid>
      <pubDate>Fri, 13 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>AWS Emulator</category>
      <category>LocalStack</category>
      <category>Moto</category>
      <category>Floci</category>
      <category>S3</category>
      <category>Testing</category>
      <category>Docker</category>
      <category>GitHub Actions</category>
      <category>Python</category>
      <category>Presigned URLs</category>
      <category>OTA Updates</category>
      <description>Set up AWS&#39;s free emulator in Docker, on Linux and in GitHub Actions, point boto3 at it, and see what it caught in my S3 tests that LocalStack 4.12 missed.</description>
      <content:encoded><![CDATA[<p>AWS released <a href="https://github.com/aws/aws-emulator"  target="_blank" rel="noreferrer">AWS Emulator</a> on 11 November 2026. It&rsquo;s a local copy of the AWS APIs that you run on your own machine. It&rsquo;s free, it needs no AWS account, the code is public domain (CC0), and it covers more than 200 services. It arrived seven months after LocalStack started requiring an account and an auth token for its current image, so a lot of people are looking at it as the free LocalStack alternative they&rsquo;ve been waiting for.</p>
<p>This post is mostly the setup: Docker on a Mac or Windows machine, the Linux package, the environment variables that point the AWS CLI and boto3 at it, Docker Compose, pytest and GitHub Actions. I ran every command here myself, on version 1.0.342. After the setup comes what it got right and wrong when I pointed the S3 tests from two of my own projects at it, side by side with LocalStack 4.12, Moto and Floci. The two projects are the upload flow in my marketplace app, and the bucket a self-hosted Codemagic Patch server keeps OTA bundles in.</p>

<h2 class="relative group">AWS Emulator in one minute
    <div id="aws-emulator-in-one-minute" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#aws-emulator-in-one-minute" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>AWS doesn&rsquo;t publish a container image. You build one from a nine-line Dockerfile that installs their Linux package, run it on port 9297, and set a handful of environment variables. From then on your SDK talks to the emulator instead of AWS, and your code doesn&rsquo;t change:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">docker build -t aws-emulator .           <span class="c1"># the Dockerfile is below</span>
</span></span><span class="line"><span class="cl">mkdir -p ~/.aws/emulator
</span></span><span class="line"><span class="cl">docker run --rm -d --name aws-emulator -p 127.0.0.1:9297:9297 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">HOME</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">&#34;</span> -v <span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator:</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  aws-emulator -p <span class="m">9297</span> --telemetry off
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ENDPOINT_URL</span><span class="o">=</span>https://127.0.0.1:9297
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SHARED_CREDENTIALS_FILE</span><span class="o">=</span>~/.aws/emulator/9297/emulator_credentials
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CONFIG_FILE</span><span class="o">=</span>~/.aws/emulator/9297/config
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CA_BUNDLE</span><span class="o">=</span>~/.aws/emulator/9297/ca.pem
</span></span><span class="line"><span class="cl">aws sts get-caller-identity              <span class="c1"># &#34;Account&#34;: &#34;000000000000&#34;</span></span></span></code></pre></div></div>
<figure><video
    class="my-0 rounded-md"
    autoplay loop muted playsinline preload="metadata"
    poster="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-setup.jpg" style="background:url('data:image/webp;base64,UklGRloAAABXRUJQVlA4IE4AAACQBACdASogABIAPw1yskwmIyIhKAqoxiGJ6QDG9BPdK4I2hDwC1QaVpF521UAA/ui2Gzha/NBVWRJqzq2oTgehY/P2cqAeQAizoM8AAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="Terminal: the Dockerfile, starting the AWS Emulator container, a health check answering ok, aws sts get-caller-identity returning account 000000000000, an S3 upload, and a CORS preflight answered with HTTP 501">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-setup.mp4" type="video/mp4">
  </video><figcaption>The whole setup, recorded on my Mac: start, health check, the emulator&rsquo;s account, an S3 upload, and the one thing it can&rsquo;t do yet (a CORS preflight answers 501, more on that below).</figcaption>
</figure>

<p>It&rsquo;s ready in about half a second and idles at about 175 MiB of memory. Everything lives in memory, so stopping the container wipes it.</p>

<h2 class="relative group">Do you need an account? (LocalStack vs AWS Emulator)
    <div id="do-you-need-an-account-localstack-vs-aws-emulator" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#do-you-need-an-account-localstack-vs-aws-emulator" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>No. AWS Emulator, Moto and Floci all run without signing up for anything. AWS Emulator doesn&rsquo;t even want your AWS credentials: it makes its own access key every time it starts and only accepts requests signed with it.</p>
<p>LocalStack is the one that changed. Since March 2026 its current image needs a <code>LOCALSTACK_AUTH_TOKEN</code>, which comes from a free account (their free plan is aimed at hobby and non-commercial use) or a paid one. Without a token the container stops at startup. I tried both: the image tagged <code>latest</code> (2026.10.7, built 11 November) quits with exit code 55, while 4.12, the last community release and the version LocalStack itself suggests pinning, still starts without one.</p>
<figure><video
    class="my-0 rounded-md"
    autoplay loop muted playsinline preload="metadata"
    poster="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/localstack-needs-account.jpg" style="background:url('data:image/webp;base64,UklGRlYAAABXRUJQVlA4IEoAAABQBACdASogABIAPw10pUymIyGhKA1QxiGJ6QDJTPgAtfKr5WoM7/AUVx4AAP7ozNYDExNyKOZLAQjfzKlQvzdAcPxaBgAGLrjAAA==') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="Terminal: localstack/localstack:latest exits with &#39;License activation failed&#39; and exit code 55, then localstack/localstack:4.12 starts and its health check reports version 4.12.0 with S3 available">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/localstack-needs-account.mp4" type="video/mp4">
  </video><figcaption>LocalStack&rsquo;s current image without a token, then the pinned 4.12. Pinning works today, but 4.12 won&rsquo;t get fixes.</figcaption>
</figure>

<p>One thing AWS Emulator does want is telemetry, which is on by default. AWS says it never includes your requests, code or credentials, but if you&rsquo;d rather it sent nothing, there are three switches: <code>--telemetry off</code> on the command line, <code>EMULATOR_TELEMETRY=off</code> in the environment, or this in <code>~/.aws/emulator/config.toml</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">telemetry</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">enabled</span> <span class="p">=</span> <span class="kc">false</span></span></span></code></pre></div></div>

<h2 class="relative group">Set up AWS Emulator with Docker on macOS or Windows
    <div id="set-up-aws-emulator-with-docker-on-macos-or-windows" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#set-up-aws-emulator-with-docker-on-macos-or-windows" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Docker is the route on a Mac or a Windows machine. Put this in an empty folder as <code>Dockerfile</code>. It&rsquo;s AWS&rsquo;s own, from <a href="https://docs.aws.amazon.com/emulator/latest/userguide/getting-started-docker.html"  target="_blank" rel="noreferrer">their Docker guide</a>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="s">public.ecr.aws/amazonlinux/amazonlinux:2023-minimal</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">ADD</span> https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator-x86_64.rpm /tmp/aws-emulator.rpm<span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">RUN</span> rpm -i /tmp/aws-emulator.rpm <span class="o">&amp;&amp;</span> rm /tmp/aws-emulator.rpm <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="o">&amp;&amp;</span> mkdir -p /home/emulator <span class="o">&amp;&amp;</span> chown <span class="m">1000</span> /home/emulator<span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">USER</span><span class="w"> </span><span class="s">1000</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">ENV</span> <span class="nv">HOME</span><span class="o">=</span>/home/emulator<span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">EXPOSE</span><span class="w"> </span><span class="s">9297</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">ENTRYPOINT</span> <span class="p">[</span><span class="s2">&#34;aws-emulator&#34;</span><span class="p">,</span> <span class="s2">&#34;-b&#34;</span><span class="p">,</span> <span class="s2">&#34;0.0.0.0&#34;</span><span class="p">]</span></span></span></code></pre></div></div>
<p>On Apple silicon or any other ARM machine, change <code>x86_64.rpm</code> to <code>aarch64.rpm</code>. Then build it and start it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">docker build -t aws-emulator .
</span></span><span class="line"><span class="cl">mkdir -p ~/.aws/emulator
</span></span><span class="line"><span class="cl">docker run --rm -d --name aws-emulator --memory<span class="o">=</span>2g --memory-swap<span class="o">=</span>2g <span class="se">\
</span></span></span><span class="line"><span class="cl">  -p 127.0.0.1:9297:9297 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">HOME</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">&#34;</span> -v <span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator:</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  aws-emulator -p <span class="m">9297</span> --telemetry off</span></span></code></pre></div></div>
<p>The build took 88 seconds on my Mac and the image is 1.67 GB. Three parts of that <code>docker run</code> line matter more than they look:</p>
<ul>
<li><strong><code>-p 127.0.0.1:9297:9297</code>, not <code>-p 9297:9297</code>.</strong> The short form opens the emulator to your whole network.</li>
<li><strong>The volume and <code>-e HOME</code>.</strong> At startup the emulator writes four files into <code>~/.aws/emulator/9297/</code>: its credentials, an AWS <code>config</code> file, its certificate (<code>ca.pem</code>), and a Java trust store. Mounting the folder at the same path inside the container is how your SDK on the host gets to read them.</li>
<li><strong>The port appears three times.</strong> If you change it, change it everywhere. The emulator writes its port into those files, and two containers that share a folder will overwrite each other&rsquo;s.</li>
</ul>

<h2 class="relative group">Install AWS Emulator on Linux
    <div id="install-aws-emulator-on-linux" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#install-aws-emulator-on-linux" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On Linux you don&rsquo;t need Docker. There&rsquo;s a Debian package and an RPM:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Debian / Ubuntu (use aws-emulator_arm64.deb on ARM)</span>
</span></span><span class="line"><span class="cl">curl -fsSLo aws-emulator.deb https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
</span></span><span class="line"><span class="cl">sudo apt-get install -y ./aws-emulator.deb
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Amazon Linux, Fedora, RHEL (aarch64.rpm on ARM)</span>
</span></span><span class="line"><span class="cl">sudo dnf install -y https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator-x86_64.rpm
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">aws-emulator --telemetry off</span></span></code></pre></div></div>
<p>The package is 150 MB. It runs in the foreground on <code>127.0.0.1:9297</code> and writes the same four files to <code>~/.aws/emulator/9297/</code>. Ctrl+C stops it cleanly and deletes them. For test runs, add <code>--idle-timeout 900</code>, which shuts it down after 15 minutes without a request, so a forgotten emulator doesn&rsquo;t sit there all week.</p>

<h2 class="relative group">Point the AWS CLI and boto3 at the emulator
    <div id="point-the-aws-cli-and-boto3-at-the-emulator" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#point-the-aws-cli-and-boto3-at-the-emulator" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>These are the variables. Any SDK that reads the standard AWS environment picks them up, so the code under test needs no endpoint and no keys:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">E</span><span class="o">=</span>~/.aws/emulator/9297
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ENDPOINT_URL</span><span class="o">=</span>https://127.0.0.1:9297
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SHARED_CREDENTIALS_FILE</span><span class="o">=</span><span class="nv">$E</span>/emulator_credentials
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CONFIG_FILE</span><span class="o">=</span><span class="nv">$E</span>/config
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CA_BUNDLE</span><span class="o">=</span><span class="nv">$E</span>/ca.pem
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_REGION</span><span class="o">=</span>us-east-1
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_EC2_METADATA_DISABLED</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_DISABLE_HOST_PREFIX_INJECTION</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="nb">unset</span> AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">curl -s --cacert <span class="nv">$AWS_CA_BUNDLE</span> <span class="nv">$AWS_ENDPOINT_URL</span>/__admin/health   <span class="c1"># ok</span>
</span></span><span class="line"><span class="cl">aws sts get-caller-identity                                          <span class="c1"># 000000000000</span></span></span></code></pre></div></div>
<p>Then plain boto3 just works:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s2">&#34;s3&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">s3</span><span class="o">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="s2">&#34;uploads&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">s3</span><span class="o">.</span><span class="n">list_buckets</span><span class="p">()[</span><span class="s2">&#34;Buckets&#34;</span><span class="p">])</span></span></span></code></pre></div></div>
<p>The <code>unset</code> line is the one to keep. Environment variables beat the credentials file, so if your real AWS keys are still exported the SDK signs with them, the emulator refuses the request, and you lose ten minutes wondering why. <code>aws sts get-caller-identity</code> is a quick sanity check: the emulator is always account <code>000000000000</code>. If you see your own account number, you&rsquo;re talking to AWS.</p>
<p>Some other things I hit, or nearly did:</p>
<ul>
<li><strong>Keep <code>127.0.0.1</code> in the endpoint, not <code>localhost</code>.</strong> With an IP address the SDKs put the bucket name in the path. With <code>localhost</code>, the JavaScript, Java and Go SDKs move it into the host name (<code>uploads.localhost</code>), and the emulator doesn&rsquo;t accept that.</li>
<li><strong>It&rsquo;s HTTPS only, with a new certificate every start.</strong> boto3, the AWS CLI and Go read <code>AWS_CA_BUNDLE</code>. Node needs <code>NODE_EXTRA_CA_CERTS</code> pointing at the same <code>ca.pem</code>, and Java uses <code>truststore.p12</code> through <code>JAVA_TOOL_OPTIONS</code>. <code>AWS_CA_BUNDLE</code> replaces the trusted certificates for that process, so a test that also calls a real HTTPS API needs a combined bundle.</li>
<li><strong>There&rsquo;s no reset.</strong> No admin route empties it, so give every test its own bucket name, or restart the emulator between suites.</li>
<li><strong>The keys change on every start.</strong> Don&rsquo;t copy them into a <code>.env</code> file. Point at the credentials file.</li>
<li><strong>If your app builds its own client with an explicit endpoint</strong> (django-storages&rsquo; <code>AWS_S3_ENDPOINT_URL</code>, for example), set that from <code>AWS_ENDPOINT_URL</code> too, or that one client keeps talking to the real service.</li>
</ul>

<h2 class="relative group">Run AWS Emulator with Docker Compose
    <div id="run-aws-emulator-with-docker-compose" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#run-aws-emulator-with-docker-compose" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For a project that already has a <code>compose.yaml</code>, this service builds the image from the Dockerfile above, keeps the emulator&rsquo;s files in the project folder instead of your home directory, and has a health check, so <code>docker compose up --wait</code> returns once it&rsquo;s ready:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">aws</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">build</span><span class="p">:</span><span class="w"> </span><span class="l">./aws-emulator           </span><span class="w"> </span><span class="c"># the folder with the Dockerfile</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;-p&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;9297&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;--telemetry&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;off&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;127.0.0.1:9297:9297&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">HOME</span><span class="p">:</span><span class="w"> </span><span class="l">${PWD}/.aws-emulator-home</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">./.aws-emulator-home/.aws/emulator:${PWD}/.aws-emulator-home/.aws/emulator</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mem_limit</span><span class="p">:</span><span class="w"> </span><span class="l">2g</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">healthcheck</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">test</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;CMD&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;curl&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;-sf&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;--cacert&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;${PWD}/.aws-emulator-home/.aws/emulator/9297/ca.pem&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">             </span><span class="s2">&#34;https://127.0.0.1:9297/__admin/health&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">interval</span><span class="p">:</span><span class="w"> </span><span class="l">1s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">retries</span><span class="p">:</span><span class="w"> </span><span class="m">30</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">mkdir -p .aws-emulator-home/.aws/emulator
</span></span><span class="line"><span class="cl">docker compose up -d --wait aws       <span class="c1"># healthy in about 5 s with the image already built</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">E</span><span class="o">=</span><span class="nv">$PWD</span>/.aws-emulator-home/.aws/emulator/9297   <span class="c1"># then the variables from above</span></span></span></code></pre></div></div>
<p>Add <code>.aws-emulator-home/</code> to <code>.gitignore</code>. It holds that run&rsquo;s keys.</p>

<h2 class="relative group">Use AWS Emulator from pytest
    <div id="use-aws-emulator-from-pytest" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#use-aws-emulator-from-pytest" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>There&rsquo;s nothing emulator-specific in the tests themselves. What I add is a guard, because the failure I worry about isn&rsquo;t a red test. It&rsquo;s a green one that ran against a real bucket:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># conftest.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd"><span>@</span>pytest.fixture</span><span class="p">(</span><span class="n">scope</span><span class="o">=</span><span class="s2">&#34;session&#34;</span><span class="p">,</span> <span class="n">autouse</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">only_against_the_emulator</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;AWS_ENDPOINT_URL&#34;</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">url</span><span class="o">.</span><span class="n">startswith</span><span class="p">((</span><span class="s2">&#34;https://127.0.0.1&#34;</span><span class="p">,</span> <span class="s2">&#34;http://127.0.0.1&#34;</span><span class="p">)):</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;AWS_ENDPOINT_URL is </span><span class="si">{</span><span class="n">url</span><span class="si">!r}</span><span class="s2">; refusing to run against real AWS&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd"><span>@</span>pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">s3</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s2">&#34;s3&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd"><span>@</span>pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">bucket</span><span class="p">(</span><span class="n">s3</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;test-</span><span class="si">{</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span><span class="o">.</span><span class="n">hex</span><span class="p">[:</span><span class="mi">12</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>   <span class="c1"># no reset route, so never reuse a name</span>
</span></span><span class="line"><span class="cl">    <span class="n">s3</span><span class="o">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">name</span></span></span></code></pre></div></div>
<p>A test then reads like any other:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_presigned_upload_rejects_the_wrong_content_type</span><span class="p">(</span><span class="n">s3</span><span class="p">,</span> <span class="n">bucket</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;put_object&#34;</span><span class="p">,</span> <span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">300</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">Params</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Bucket&#34;</span><span class="p">:</span> <span class="n">bucket</span><span class="p">,</span> <span class="s2">&#34;Key&#34;</span><span class="p">:</span> <span class="s2">&#34;media/a.jpg&#34;</span><span class="p">,</span> <span class="s2">&#34;ContentType&#34;</span><span class="p">:</span> <span class="s2">&#34;image/jpeg&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="sa">b</span><span class="s2">&#34;&lt;html&gt;&#34;</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;text/html&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">                     <span class="n">verify</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;AWS_CA_BUNDLE&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">403</span></span></span></code></pre></div></div>
<p>That one is from my marketplace app&rsquo;s upload flow, and it&rsquo;s a good example of why the emulator you pick matters: three of the five I tried return 200 for it. More on that below. If you only run the affected tests on each change, <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/" >pytest-testmon</a> works with this unchanged.</p>

<h2 class="relative group">Run AWS Emulator in GitHub Actions
    <div id="run-aws-emulator-in-github-actions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#run-aws-emulator-in-github-actions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On a GitHub runner, skip Docker and install the Debian package. The background process from one step keeps running for the rest of the job:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">test</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">astral-sh/setup-uv@v10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Start AWS Emulator</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          curl -fsSLo aws-emulator.deb https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
</span></span></span><span class="line"><span class="cl"><span class="sd">          sudo apt-get install -y ./aws-emulator.deb
</span></span></span><span class="line"><span class="cl"><span class="sd">          aws-emulator --telemetry off --idle-timeout 900 &gt; emulator.log 2&gt;&amp;1 &amp;
</span></span></span><span class="line"><span class="cl"><span class="sd">          E=$HOME/.aws/emulator/9297
</span></span></span><span class="line"><span class="cl"><span class="sd">          for i in $(seq 30); do
</span></span></span><span class="line"><span class="cl"><span class="sd">            curl -sf --cacert $E/ca.pem https://127.0.0.1:9297/__admin/health &amp;&amp; break
</span></span></span><span class="line"><span class="cl"><span class="sd">            sleep 1
</span></span></span><span class="line"><span class="cl"><span class="sd">          done
</span></span></span><span class="line"><span class="cl"><span class="sd">          {
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_ENDPOINT_URL=https://127.0.0.1:9297&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_SHARED_CREDENTIALS_FILE=$E/emulator_credentials&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_CONFIG_FILE=$E/config&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_CA_BUNDLE=$E/ca.pem&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_REGION=us-east-1&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_EC2_METADATA_DISABLED=true&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_DISABLE_HOST_PREFIX_INJECTION=true&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          } &gt;&gt; &#34;$GITHUB_ENV&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv run pytest</span></span></span></code></pre></div></div>
<p>I ran the same steps in a fresh <code>ubuntu:24.04</code> container (with the arm64 package, since my Mac is ARM). The install took 29 seconds, nearly all of it the 150 MB download, and the health check answered within 4 seconds. Don&rsquo;t add real AWS credentials to this job at all. That&rsquo;s the cheapest way to be sure nothing in it reaches AWS.</p>
<p><code>LTS/latest</code> is whatever was released last, and as you&rsquo;ll see below that changes several times a day. If you want CI to be reproducible, AWS has a page on <a href="https://docs.aws.amazon.com/emulator/latest/userguide/versioning.html"  target="_blank" rel="noreferrer">installing a specific version</a>. For how I lay out the rest of a pipeline, see <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-docker-compose-ci-github-actions/" >the GitHub Actions pipelines I run now</a>.</p>

<h2 class="relative group">Moving from LocalStack
    <div id="moving-from-localstack" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#moving-from-localstack" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If your tests already run against LocalStack, most of the move is configuration:</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>LocalStack</th>
					<th>AWS Emulator</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Endpoint</td>
					<td><code>http://localhost:4566</code></td>
					<td><code>https://127.0.0.1:9297</code> plus <code>AWS_CA_BUNDLE</code></td>
			</tr>
			<tr>
					<td>Credentials</td>
					<td>anything (<code>test</code> / <code>test</code>)</td>
					<td>the file it writes each run, or <code>--auth off</code></td>
			</tr>
			<tr>
					<td>Account ID</td>
					<td><code>000000000000</code></td>
					<td><code>000000000000</code></td>
			</tr>
			<tr>
					<td>Setup scripts</td>
					<td>init hooks in <code>/etc/localstack/init/ready.d</code></td>
					<td>none: create resources in fixtures, after the health check</td>
			</tr>
			<tr>
					<td>Image</td>
					<td><code>localstack/localstack</code> (token needed after 4.12)</td>
					<td>build your own from AWS&rsquo;s package</td>
			</tr>
	</tbody>
</table>
<p>The init hooks are the part that takes real work if you use them a lot. I didn&rsquo;t, because each test makes its own bucket. And nothing survives a restart: the emulator keeps everything in memory.</p>

<h2 class="relative group">What AWS Emulator caught in my S3 tests, and what it missed
    <div id="what-aws-emulator-caught-in-my-s3-tests-and-what-it-missed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-aws-emulator-caught-in-my-s3-tests-and-what-it-missed" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The setup is the easy part. What I wanted to know was whether it behaves like S3 where my code depends on S3&rsquo;s behaviour. I wrote 12 checks, each asserting what real S3 does according to AWS&rsquo;s documentation:</p>
<ul>
<li><strong>Six from the marketplace app&rsquo;s upload flow.</strong> The app hands the browser a presigned URL, the browser PUTs the photo straight into the bucket (so the bucket needs CORS), and an <code>ObjectCreated</code> event tells the backend the file has arrived. That&rsquo;s the flow from <a href="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/" >my post on object notifications</a>, with SQS standing in for the webhook.</li>
<li><strong>Six from a Codemagic Patch bucket.</strong> Patch is an OTA update server for React Native apps (<a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-13-moving-to-codemagic-patch/" >how I moved to it</a>). Its bucket is public, so phones can download bundles, except for an <code>_internal/</code> prefix where uploads wait to be published, and a bucket policy keeps that prefix private.</li>
</ul>
<p>I ran them against AWS Emulator 1.0.342 (once as it ships and once with <code>--auth off</code>), LocalStack 4.12 (as it ships, and with <code>S3_SKIP_SIGNATURE_VALIDATION=0</code>), Moto 5.2.6 in server mode, Floci 1.5.5, and Silo, the MinIO fork that Patch&rsquo;s self-hosted bundle runs.</p>
<figure><video
    class="my-0 rounded-md"
    autoplay loop muted playsinline preload="metadata"
    poster="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/s3-emulator-test-run.jpg" style="background:url('data:image/webp;base64,UklGRk4AAABXRUJQVlA4IEIAAAAQBACdASogABIAPw10rkymIyIhKAqoxiGJ6QDM0PgAtf/5Tl8J29EIQAD+6MzcUMCFo8DULSYQXf8FTN9rmZMAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="Terminal: a script runs the same 12 S3 checks against seven emulator setups and prints a pass/fail matrix; every setup ends between 6 and 8 out of 12">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/s3-emulator-test-run.mp4" type="video/mp4">
  </video><figcaption>The real run. Nothing passed all 12.</figcaption>
</figure>

<p>No setup passed all 12; they all landed between 6 and 8. The totals matter less than which checks each one missed, so here are the ones that would have hurt me, worst first.</p>
<p><strong>AWS Emulator is the strictest about permissions.</strong> Take the presigned URL my app signs for one JPEG. Against LocalStack 4.12 as it ships, Moto and Floci, I could use that same URL to upload an HTML file and get a 200 back. On a bucket that serves public files, that&rsquo;s a stored-XSS problem waiting to happen. AWS Emulator refused it with a 403, as S3 does, and it refused the URL again once it had expired. (LocalStack 4.12 catches both if you set <code>S3_SKIP_SIGNATURE_VALIDATION=0</code>, and Floci catches the expiry but not the swapped file type.) It was also the only one that knew new S3 buckets have Block Public Access switched on, so a public bucket policy is refused until you turn that off, and the only one that applied an explicit <code>Deny</code> to the server&rsquo;s own signed requests as well as to strangers. These are exactly the mistakes that pass every test and then fail on the first real deploy.</p>
<p><strong>It can&rsquo;t test anything a browser does yet.</strong> A CORS preflight (the <code>OPTIONS</code> request a browser sends before a cross-origin PUT) gets <code>501 Not Implemented</code>. With authentication on, which is the default, it also refuses every unsigned request, so an anonymous download from a public bucket gets a 403 even when the policy allows it. Both gaps are already filed in the repository. The project keeps its known gaps as files under <code>issues/things-that-do-not-work/</code>, and on 13 November there were 6,840 of them, with 13,409 already moved to <code>things-that-now-work/</code>.</p>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="AWS Emulator&rsquo;s issue file for S3, titled &ldquo;7 net-new causes — CORS preflight answers 501…&rdquo;, in the things-that-do-not-work folder on GitHub"
    width="1400"
    height="804"
    src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_33033b091b284c0d.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_33033b091b284c0d.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_f0294a8a0a022038.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_925176ba1af6cb05.webp"></span><figcaption>The CORS gap, filed by the project&rsquo;s own issue-finding agent on 9 November.</figcaption></figure>
</p>
<p><strong><code>--auth off</code> swaps one half for the other.</strong> With authentication off, anonymous reads work and the bucket policy is evaluated properly, so all six Patch checks pass. That&rsquo;s the only setup that managed it. But presigned URLs are no longer checked either, so the upload checks fall to two. Running two emulators on different ports, one with auth and one without, covers both halves, though it&rsquo;s one more thing to explain to whoever reads the CI file next.</p>
<p><strong>It&rsquo;s moving fast.</strong> Releases v1.0.290 to v1.0.342 came out in three days, each a long list of fixes (AWS says coding agents built it, and the pace shows), and one of the 11 November fixes was &ldquo;presigned GET refused with 403&rdquo; in S3. Anything in my table could be different by next week, in either direction. Pin a version in CI and move it deliberately.</p>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="The aws/aws-emulator releases page on GitHub: v1.0.342 marked Latest, released 3 hours earlier, with nine more releases listed in the sidebar"
    width="1600"
    height="1125"
    src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_e39a0eec6b783a7a.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_e39a0eec6b783a7a.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_df197ee02b7c435c.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_59bd3abd9b9aaf92.webp"></span><figcaption>Ten releases in three days. Pin one.</figcaption></figure>
</p>

<h2 class="relative group">The Patch bucket: one policy condition that only matters on real S3
    <div id="the-patch-bucket-one-policy-condition-that-only-matters-on-real-s3" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-patch-bucket-one-policy-condition-that-only-matters-on-real-s3" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This was the part I didn&rsquo;t expect. Patch&rsquo;s self-hosted bundle keeps everything in one bucket, and the policy it applies to its bundled storage is the obvious one: anyone can read, except under <code>_internal/</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;Version&#34;</span><span class="p">:</span> <span class="s2">&#34;2012-10-17&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;Statement&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;Effect&#34;</span><span class="p">:</span> <span class="s2">&#34;Allow&#34;</span><span class="p">,</span> <span class="nt">&#34;Principal&#34;</span><span class="p">:</span> <span class="s2">&#34;*&#34;</span><span class="p">,</span> <span class="nt">&#34;Action&#34;</span><span class="p">:</span> <span class="s2">&#34;s3:GetObject&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;Resource&#34;</span><span class="p">:</span> <span class="s2">&#34;arn:aws:s3:::example-ota/*&#34;</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;Effect&#34;</span><span class="p">:</span> <span class="s2">&#34;Deny&#34;</span><span class="p">,</span> <span class="nt">&#34;Principal&#34;</span><span class="p">:</span> <span class="s2">&#34;*&#34;</span><span class="p">,</span> <span class="nt">&#34;Action&#34;</span><span class="p">:</span> <span class="s2">&#34;s3:GetObject&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;Resource&#34;</span><span class="p">:</span> <span class="s2">&#34;arn:aws:s3:::example-ota/_internal/*&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>On Silo, signed with the admin credentials, that does what it says: strangers can&rsquo;t read <code>_internal/</code>, and the server&rsquo;s own copy from <code>_internal/</code> into the public prefix (the &ldquo;publish&rdquo; step) goes through. On real S3 an explicit <code>Deny</code> with <code>&quot;Principal&quot;: &quot;*&quot;</code> applies to everyone, including your own credentials, so the same policy would block the server&rsquo;s own publish step. AWS Emulator refused that copy with a 403, which is what AWS&rsquo;s policy evaluation rules say S3 does.</p>
<p>Patch&rsquo;s maintainers clearly know this, because their <a href="https://github.com/codemagic-ci-cd/codemagic-patch/blob/main/deploy/selfhost/aws-s3-bucket-policy.example.json"  target="_blank" rel="noreferrer">example policy for AWS S3</a> adds one condition to the <code>Deny</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="s2">&#34;Condition&#34;</span><span class="err">:</span> <span class="p">{</span> <span class="nt">&#34;StringNotEquals&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;aws:PrincipalAccount&#34;</span><span class="p">:</span> <span class="s2">&#34;YOUR_AWS_ACCOUNT_ID&#34;</span> <span class="p">}</span> <span class="p">}</span></span></span></code></pre></div></div>
<p>With that line, AWS Emulator (with <code>--auth off</code>, so the anonymous half could be tested too) and Moto both let the worker publish while still refusing anonymous reads of <code>_internal/</code>. But only AWS Emulator caught the version without it. Moto, LocalStack 4.12 and Floci all let the copy through either way, so every one of them would have let me ship the broken policy. That&rsquo;s the clearest argument I found for testing policies against the strictest emulator available. Copy a policy from a MinIO setup to S3 without that condition and nothing complains until the first real publish.</p>
<p>(My own Patch server stores bundles on Tigris, which has no bucket policies at all, so I keep <code>_internal/</code> in a separate private bucket. That&rsquo;s what <code>S3_INTERNAL_BUCKET</code> is for in Patch&rsquo;s configuration. I also write paid guest posts about Patch for Codemagic. This isn&rsquo;t one of them, and nobody paid for it.)</p>

<h2 class="relative group">Which one I&rsquo;d use now
    <div id="which-one-id-use-now" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#which-one-id-use-now" aria-label="Anchor">#</a>
    </span>
    
</h2>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Ready in</th>
					<th>Idle memory</th>
					<th>Image</th>
					<th>Account</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>AWS Emulator 1.0.342</td>
					<td>0.4–0.5 s</td>
					<td>176 MiB</td>
					<td>1.67 GB (you build it)</td>
					<td>none</td>
			</tr>
			<tr>
					<td>LocalStack 4.12</td>
					<td>0.9–2.2 s</td>
					<td>90 MiB</td>
					<td>1.89 GB</td>
					<td>none for 4.12, token after</td>
			</tr>
			<tr>
					<td>Moto 5.2.6 (server)</td>
					<td>0.7–0.8 s</td>
					<td>45 MiB</td>
					<td>560 MB</td>
					<td>none</td>
			</tr>
			<tr>
					<td>Floci 1.5.5</td>
					<td>0.3–0.4 s</td>
					<td>11 MiB</td>
					<td>292 MB</td>
					<td>none</td>
			</tr>
	</tbody>
</table>
<p>Measured on my Mac in Docker, three cold starts each. None of them is slow enough to matter.</p>
<p>For backend tests I&rsquo;d start with AWS Emulator, because the mistakes it catches (permissions, signatures, Block Public Access) are the ones that otherwise only show up in production, and it&rsquo;s the only one in this list that&rsquo;s free, current and made by the people who make S3. Until the CORS preflight is implemented, I&rsquo;d keep one small job for the browser-facing checks (the preflight, and an anonymous download from the public prefix) on Floci. LocalStack 4.12 with strict signatures passes them too, but it&rsquo;s frozen and won&rsquo;t get fixes. If your tests are Python and you&rsquo;d rather not run a container, Moto&rsquo;s in-process mocks are still the lightest option. Just don&rsquo;t rely on it for presigned URL checks.</p>
<p>The 12 checks are short enough to write for your own buckets in an afternoon, and I&rsquo;d recommend doing that before you trust any emulator, this one included. Each tool in the table failed different checks, and those are the gaps your tests would quietly pass over.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>pytest-testmon: Run Only the Tests Affected by Your Code Change</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/</guid>
      <pubDate>Mon, 26 Oct 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Python</category>
      <category>Django</category>
      <category>pytest</category>
      <category>Testing</category>
      <category>Test Impact Analysis</category>
      <category>CI/CD</category>
      <category>GitHub Actions</category>
      <category>Claude Code</category>
      <category>AI Agents</category>
      <category>Efficiency</category>
      <description>pytest-testmon runs only the tests a change can reach, from a coverage record, not a guess. Adding it to a 2,248-test Django suite, the wrapper it needs, and a Python 3.14 trap.</description>
      <content:encoded><![CDATA[<p><a href="https://www.testmon.org/"  target="_blank" rel="noreferrer">pytest-testmon</a> is a pytest plugin that runs only the tests your change can affect. It watches which lines of code each test executes, using coverage.py, and stores a checksum of every block of code a test touched. On the next run it compares those checksums with the code on disk and runs only the tests whose blocks changed, plus anything that failed last time, so it also does the job of <code>pytest --lf</code>.</p>
<p>The general technique is called test impact analysis, and testmon is a free, open-source way to get it in Python. Nothing about it depends on git: it compares against what it saw on the last run, committed or not.</p>
<p>I added it to my marketplace app&rsquo;s Python tests this week, and the reason wasn&rsquo;t speed for its own sake.</p>
<figure>
  <div class="mc-film">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRmYAAABXRUJQVlA4IFoAAADwBACdASogABIAPw1wqUqmIyGYDAQAYxDE9IAAS3igrsLc+giUXH/mz758faBcoAAA/vVl8+gZ7GYAPuvC1znoo64X/UaaVXzZOS4gxmDEyDUzHvkl6JAAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film_en.vtt" srclang="en" label="English">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film_ja.vtt" srclang="ja" label="日本語">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film_ko.vtt" srclang="ko" label="한국어">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film_vi.vtt" srclang="vi" label="Tiếng Việt">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/testmon-film_zh-cn.vtt" srclang="zh-cn" label="简体中文">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script><figcaption>This post as an eight-minute film, if you&rsquo;d rather watch than read.</figcaption>
</figure>


<h2 class="relative group">Why I added it
    <div id="why-i-added-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-i-added-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A feature request is usually small: one screen, one endpoint, one rule that should work differently. Running the whole test suite made every one of them bigger than it was. The suite is 2,248 Django tests, about three and a half minutes with four processes and 11 in one, and every edit ran all of it, though almost all of it checks things the request never touched. When something unrelated failed, or the run simply took its time, the feature waited. It was being held up by circumstances outside the feature request, and the thing holding it up was my own test suite.</p>
<p>The alternative wasn&rsquo;t better. Skip the full run, guess which test file covers the change, and now and then guess wrong. That&rsquo;s quick, but it isn&rsquo;t precise, and an untested change I ship is a feature request I&rsquo;ll be doing twice.</p>
<p>The tests I wait for should be exactly the tests the change can reach, no more and no fewer. testmon gives me that set from a record instead of a guess. The full suite still runs, in CI, where it doesn&rsquo;t hold up whatever I&rsquo;m working on next.</p>

<h2 class="relative group">How it decides what to run
    <div id="how-it-decides-what-to-run" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-it-decides-what-to-run" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>testmon splits each Python file into blocks: the module&rsquo;s top level, each class, each function, with the bodies of nested functions cut out of the block around them. It stores a checksum of each block, never the code itself. While a test runs, coverage.py records the lines it executes, and testmon notes which blocks those lines belong to. There&rsquo;s no static analysis and no guessing from imports. A test depends on exactly the blocks it ran.</p>
<p>Take a module like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># pricing.py</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">subtotal</span><span class="p">(</span><span class="n">items</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">price</span> <span class="o">*</span> <span class="n">item</span><span class="o">.</span><span class="n">quantity</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">items</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">discount</span><span class="p">(</span><span class="n">total</span><span class="p">,</span> <span class="n">code</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">code</span> <span class="o">==</span> <span class="s2">&#34;WELCOME10&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">total</span> <span class="o">*</span> <span class="mf">0.9</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">total</span></span></span></code></pre></div></div>
<p>A test that only calls <code>subtotal()</code> depends on the module&rsquo;s top level and on <code>subtotal</code>&rsquo;s body. Rewrite <code>discount()</code> however you like and testmon won&rsquo;t rerun that test, because it never executed a line of <code>discount</code>. Change one character inside <code>subtotal</code> and it runs again, even if the change can&rsquo;t alter the result. testmon doesn&rsquo;t try to be clever about side effects: if a block you executed changed, you run.</p>
<p>It also tracks what&rsquo;s around the code. The record includes the Python version and the installed packages with their versions. It doesn&rsquo;t watch environment variables: if the same code runs under two settings modules, name them with <code>--testmon-env</code> (step 8) or they share one record. Upgrade Django and everything that touched Django is affected again, which in a Django app is everything.</p>

<h2 class="relative group">What it can&rsquo;t see
    <div id="what-it-cant-see" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-it-cant-see" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>testmon tracks Python that a test executes. It doesn&rsquo;t track the other files your tests read: templates, JSON, text, anything else on disk. It doesn&rsquo;t track services reached over the network. And it can&rsquo;t pin anything that runs once before the tests start on a particular test.</p>
<p>In a Django project that means migrations, which run once when the test database is created, so no test&rsquo;s record includes them (the wrapper below rebuilds the test database when there&rsquo;s a new one); settings and start-up code, which run while pytest starts up, outside any one test; and templates and data files, since a template isn&rsquo;t Python, so editing one reruns nothing that renders it.</p>
<p>My suite has a few more of its own. Some tests read a JSON file of fixtures that the mobile app&rsquo;s tests share. Some check the built web bundle, and some check the output of a static site generated in the same repository. Change any of those and testmon will happily tell you nothing needs to run. That&rsquo;s why the how-to below wraps testmon in a script that notices these files and runs everything when one has changed.</p>

<h2 class="relative group">How to add it to a Django project
    <div id="how-to-add-it-to-a-django-project" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-to-add-it-to-a-django-project" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>These steps assume pytest and <a href="https://pytest-django.readthedocs.io/"  target="_blank" rel="noreferrer">pytest-django</a>, and uv for the dependencies. With pip, swap <code>uv add --dev</code> for <code>pip install</code> and drop <code>uv run</code>.</p>
<p><strong>1. Install it.</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">uv add --dev pytest-testmon</span></span></code></pre></div></div>
<p><strong>2. Make sure pytest can run the suite.</strong> testmon is a pytest plugin, so it does nothing under <code>python manage.py test</code>. pytest-django needs the settings module and the file patterns your tests use:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># pytest.ini</span>
</span></span><span class="line"><span class="cl"><span class="k">[pytest]</span>
</span></span><span class="line"><span class="cl"><span class="na">DJANGO_SETTINGS_MODULE</span> <span class="o">=</span> <span class="s">myproject.settings</span>
</span></span><span class="line"><span class="cl"><span class="na">python_files</span> <span class="o">=</span> <span class="s">tests.py test_*.py *_tests.py</span>
</span></span><span class="line"><span class="cl"><span class="na">python_classes</span> <span class="o">=</span> <span class="s">Test* *Tests *TestCase</span></span></span></code></pre></div></div>
<p>If the suite has only ever run under Django&rsquo;s runner, run it under pytest once and fix whatever differs before adding testmon. Otherwise you&rsquo;ll be debugging two things at once.</p>
<p><strong>3. Keep the record out of git.</strong> testmon writes <code>.testmondata</code> in the pytest root directory. It&rsquo;s a SQLite database, so you&rsquo;ll also see <code>-shm</code> and <code>-wal</code> files next to it, and the script below adds a stamp file. One pattern covers all of them:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"># .gitignore
</span></span><span class="line"><span class="cl">.testmondata*</span></span></code></pre></div></div>
<p><strong>4. Record the whole suite once.</strong> testmon can&rsquo;t select anything until it has seen every test run:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">uv run pytest --testmon</span></span></code></pre></div></div>
<p>In a fresh checkout that first run is the whole suite in one process. Here it takes about 12 minutes for 2,248 tests. From then on, the same command runs only the affected tests. With nothing changed, it runs nothing.</p>
<p>Two catches. On testmon 2.2.0 and earlier, it needs pytest&rsquo;s cache plugin: my first attempt ran with <code>-p no:cacheprovider</code> and crashed with <code>KeyError: 'lf'</code> (2.3.0 treats a missing <code>--lf</code> as unset). And on Python 3.14 or later, set <code>COVERAGE_CORE=ctrace</code> before you record. &ldquo;The bug that looked like a green run&rdquo;, further down, explains why. It&rsquo;s the most important line in the script below.</p>
<p>Update, 7 November: testmon 2.3.0, released on 2 November, forces coverage&rsquo;s C tracer itself, so on that version the export is belt and braces. Keep it if you&rsquo;re on 2.2.0 or earlier, and keep the break-something check whatever the version.</p>
<p><strong>5. Wrap it for the files it can&rsquo;t see.</strong> This is the part that makes it safe to use every day. The script keeps a stamp file from the start of the last run. If any file testmon can&rsquo;t trace is newer than the stamp, it runs everything and records afresh. A new migration only rebuilds the test database. Otherwise it lets testmon choose. A cut-down example:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/usr/bin/env bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># scripts/test-changed.sh: the tests a change can affect, or all when in doubt.</span>
</span></span><span class="line"><span class="cl"><span class="c1">#   scripts/test-changed.sh            # affected tests</span>
</span></span><span class="line"><span class="cl"><span class="c1">#   scripts/test-changed.sh --all      # everything, re-recorded</span>
</span></span><span class="line"><span class="cl"><span class="c1">#   scripts/test-changed.sh -k orders  # anything else goes to pytest</span>
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> <span class="s2">&#34;</span><span class="k">$(</span>dirname <span class="s2">&#34;</span><span class="nv">$0</span><span class="s2">&#34;</span><span class="k">)</span><span class="s2">/..&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># coverage&#39;s C tracer: from Python 3.14 coverage defaults to sys.monitoring,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># which reports a line only the first time a process runs it, so testmon</span>
</span></span><span class="line"><span class="cl"><span class="c1"># would credit each line to one test. Changing this means --all.</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">COVERAGE_CORE</span><span class="o">=</span>ctrace
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">STAMP</span><span class="o">=</span>.testmondata.started
</span></span><span class="line"><span class="cl"><span class="nv">full</span><span class="o">=</span><span class="nb">false</span>
</span></span><span class="line"><span class="cl"><span class="nv">rebuild</span><span class="o">=</span><span class="nb">false</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="o">[</span> <span class="s2">&#34;</span><span class="si">${</span><span class="nv">1</span><span class="k">:-</span><span class="si">}</span><span class="s2">&#34;</span> <span class="o">=</span> <span class="s2">&#34;--all&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    <span class="nv">full</span><span class="o">=</span>true<span class="p">;</span> <span class="nb">shift</span>
</span></span><span class="line"><span class="cl"><span class="k">elif</span> <span class="o">[</span> ! -f .testmondata <span class="o">]</span> <span class="o">||</span> <span class="o">[</span> ! -f <span class="s2">&#34;</span><span class="nv">$STAMP</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;No record yet: running everything once.&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nv">full</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="k">else</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Changed since the last run started, and not traceable per test:</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># anything that isn&#39;t Python, plus settings and conftest.</span>
</span></span><span class="line"><span class="cl">    <span class="nv">newer</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span>find myapp myproject fixtures pytest.ini <span class="se">\
</span></span></span><span class="line"><span class="cl">        -type f -newer <span class="s2">&#34;</span><span class="nv">$STAMP</span><span class="s2">&#34;</span> ! -path <span class="s1">&#39;*/migrations/*&#39;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        <span class="se">\(</span> ! -name <span class="s1">&#39;*.py&#39;</span> -o -path <span class="s1">&#39;myproject/settings*&#39;</span> -o -name <span class="s1">&#39;conftest.py&#39;</span> <span class="se">\)</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        ! -path <span class="s1">&#39;*/__pycache__/*&#39;</span> <span class="p">|</span> head -5<span class="k">)</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="o">[</span> -n <span class="s2">&#34;</span><span class="nv">$newer</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">        <span class="nb">echo</span> <span class="s2">&#34;Changed, and not traceable per test:&#34;</span><span class="p">;</span> <span class="nb">printf</span> <span class="s1">&#39;  %s\n&#39;</span> <span class="nv">$newer</span>
</span></span><span class="line"><span class="cl">        <span class="nv">full</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl">    <span class="k">fi</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># A new migration needs a fresh test database, not a full run.</span>
</span></span><span class="line"><span class="cl">    <span class="o">[</span> -n <span class="s2">&#34;</span><span class="k">$(</span>find myapp -path <span class="s1">&#39;*/migrations/*.py&#39;</span> -newer <span class="s2">&#34;</span><span class="nv">$STAMP</span><span class="s2">&#34;</span> <span class="p">|</span> head -1<span class="k">)</span><span class="s2">&#34;</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">rebuild</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Stamp the start, so an edit made while the tests run counts next time.</span>
</span></span><span class="line"><span class="cl"><span class="nv">started</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span>mktemp .testmondata.run.XXXXXX<span class="k">)</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">args</span><span class="o">=(</span>--testmon --reuse-db<span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># --create-db alone deletes the database when the run ends; with</span>
</span></span><span class="line"><span class="cl"><span class="c1"># --reuse-db as well, the rebuilt one is kept for next time.</span>
</span></span><span class="line"><span class="cl"><span class="o">[</span> <span class="s2">&#34;</span><span class="nv">$rebuild</span><span class="s2">&#34;</span> <span class="o">=</span> <span class="nb">true</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">args</span><span class="o">=(</span>--testmon --reuse-db --create-db<span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="o">[</span> <span class="s2">&#34;</span><span class="nv">$full</span><span class="s2">&#34;</span> <span class="o">=</span> <span class="nb">true</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">args</span><span class="o">=(</span>--testmon-noselect --reuse-db --create-db<span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">status</span><span class="o">=</span><span class="m">0</span>
</span></span><span class="line"><span class="cl">uv run pytest <span class="s2">&#34;</span><span class="si">${</span><span class="nv">args</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span> -q <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span> <span class="o">||</span> <span class="nv">status</span><span class="o">=</span><span class="nv">$?</span>
</span></span><span class="line"><span class="cl"><span class="c1"># 0: passed, or nothing selected. 1: some failed (still recorded).</span>
</span></span><span class="line"><span class="cl"><span class="c1"># 5: nothing collected at all.</span>
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="s2">&#34;</span><span class="nv">$status</span><span class="s2">&#34;</span> in 0<span class="p">|</span>1<span class="p">|</span>5<span class="o">)</span> mv <span class="s2">&#34;</span><span class="nv">$started</span><span class="s2">&#34;</span> <span class="s2">&#34;</span><span class="nv">$STAMP</span><span class="s2">&#34;</span> <span class="p">;;</span> *<span class="o">)</span> rm -f <span class="s2">&#34;</span><span class="nv">$started</span><span class="s2">&#34;</span> <span class="p">;;</span> <span class="k">esac</span>
</span></span><span class="line"><span class="cl"><span class="o">[</span> <span class="s2">&#34;</span><span class="nv">$status</span><span class="s2">&#34;</span> <span class="o">=</span> <span class="m">5</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nb">exit</span> <span class="m">0</span>
</span></span><span class="line"><span class="cl"><span class="nb">exit</span> <span class="s2">&#34;</span><span class="nv">$status</span><span class="s2">&#34;</span></span></span></code></pre></div></div>
<p>Two details in there are worth explaining. <code>--reuse-db</code> is pytest-django&rsquo;s flag for keeping the test database between runs, which matters once a run takes seconds: creating it again each time can take longer than the tests themselves. A full run, or one after a new migration, adds <code>--create-db</code> so the database has the current schema. Keep <code>--reuse-db</code> next to it: <code>--create-db</code> on its own makes pytest-django delete the database when the run ends, so the next run would build it all over again.</p>
<p>And when testmon deselects everything, the run exits 0, so &ldquo;nothing changed&rdquo; counts as a pass. Exit code 5 is pytest&rsquo;s &ldquo;no tests collected&rdquo;, which the script treats as fine too.</p>
<p>Notice what&rsquo;s not in the list: <code>pyproject.toml</code> and the lock file. My first version ran everything whenever the lock file changed, which made the slow fallback fire on most days and the whole thing worse than before. testmon already tracks installed package versions itself, so it doesn&rsquo;t need the help.</p>
<p><strong>6. Give it its own database and cache.</strong> If other things run tests on the same machine, like a second terminal, a coding agent or <code>manage.py test</code>, point the script at its own test database and its own cache number so a kept database never meets someone else&rsquo;s run:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">DATABASE_URL</span><span class="o">=</span><span class="s2">&#34;postgresql://myapp:myapp@localhost:5432/myapp_changed&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">REDIS_URL</span><span class="o">=</span><span class="s2">&#34;redis://localhost:6379/8&#34;</span></span></span></code></pre></div></div>
<p><strong>7. Empty that cache before each run.</strong> I learnt this one the hard way. Subset runs started failing with <code>KeyError</code>s that full runs didn&rsquo;t. Some category labels are cached with no expiry, and a test database kept from one run met a cache still warm from an earlier one. Clearing the cache at the top of the script fixed it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">uv run python -c <span class="s2">&#34;import os, redis; redis.Redis.from_url(os.environ[&#39;REDIS_URL&#39;]).flushdb()&#34;</span></span></span></code></pre></div></div>
<p>That&rsquo;s testmon doing what its docs say it will: running a different subset each time exposes tests that only passed because of what ran before them. The fix belongs in the test setup, not in testmon.</p>
<p><strong>8. Know the flags for when selection is wrong.</strong></p>
<ul>
<li><code>--testmon-noselect</code> runs every test but puts the likely failures first. testmon switches to it by itself when you also pass <code>-k</code>, <code>-m</code>, <code>--lf</code> or a test&rsquo;s node ID.</li>
<li><code>--testmon-forceselect</code> runs only affected tests that also match your <code>-k</code> or <code>-m</code>.</li>
<li><code>--testmon-nocollect</code> uses the existing record but doesn&rsquo;t add to it. It&rsquo;s forced under a debugger or when coverage is already running.</li>
<li><code>--no-testmon</code> turns it off for one run, for when it&rsquo;s in <code>addopts</code>.</li>
<li><code>--testmon-env=&lt;name&gt;</code> keeps separate records in one file, for a suite you run under two settings modules.</li>
</ul>
<p>If the project has a JavaScript side, Jest has its own version built in. <code>jest --changedSince=main</code> runs the tests that import anything changed since <code>main</code>, following imports rather than coverage. I added it to the app&rsquo;s <code>package.json</code> as <code>npm run test:changed</code> so both halves have the same habit. An edit to one utility module selected 7 of the app&rsquo;s 121 test files.</p>

<h2 class="relative group">What it saves
    <div id="what-it-saves" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-it-saves" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I timed the wrapper once it was in place, start-up included, after fixing the bug in the next section:</p>
<table>
	<thead>
			<tr>
					<th>What changed</th>
					<th style="text-align: right">Tests run</th>
					<th style="text-align: right">Time</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Nothing</td>
					<td style="text-align: right">0</td>
					<td style="text-align: right">1.0 s</td>
			</tr>
			<tr>
					<td>One edit to the code behind a home-screen banner</td>
					<td style="text-align: right">17</td>
					<td style="text-align: right">8.5 s</td>
			</tr>
			<tr>
					<td>Reverting that edit</td>
					<td style="text-align: right">17</td>
					<td style="text-align: right">8.7 s</td>
			</tr>
			<tr>
					<td>The whole suite, for comparison (4 processes)</td>
					<td style="text-align: right">2,248</td>
					<td style="text-align: right">about 3½ min</td>
			</tr>
	</tbody>
</table>
<p>For an everyday edit that&rsquo;s about 25 times faster than the parallel suite, and when nothing has changed it&rsquo;s about two hundred times faster.</p>
<p>The record says the same thing more broadly. Of the 307 non-test Python files the suite runs, a change to the median one reaches 22 tests, about 8 seconds of test time. A few core modules sit at the other end: almost every test runs them, so a change there reaches around 1,500 tests, and testmon reruns most of the suite. That&rsquo;s the right answer, not a failure. Those figures are upper bounds per file: testmon works per block, so changing one function selects only the tests that ran that function.</p>
<p>The cost shows up when the wrapper falls back. A full run is the same single-process recording as the first one, about 12 minutes, which is slower than the parallel suite it replaces. It&rsquo;s single-process only because I haven&rsquo;t tried testmon under pytest-xdist yet, which it has supported since version 1.4; that&rsquo;s the obvious next thing to measure. That&rsquo;s why the list of files that trigger it should be as short as it can honestly be.</p>

<h2 class="relative group">The bug that looked like a green run
    <div id="the-bug-that-looked-like-a-green-run" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-bug-that-looked-like-a-green-run" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My first numbers were better than these: one edit, 6 tests, 3.3 seconds. They were wrong, and nothing on screen said so.</p>
<p>From Python 3.14, coverage.py uses a new core by default, built on the interpreter&rsquo;s <code>sys.monitoring</code>. That core reports a line only the first time a process runs it. testmon builds its record of which test ran which line from coverage, so each line was credited to the first test that happened to run it, and to no other. Edit a line that eight tests share, and testmon selects one of them.</p>
<p>It was caught on a second, smaller project of mine that copied this setup. Part of adopting it there was breaking a line on purpose and comparing the results, and the numbers didn&rsquo;t match. Back in this suite I ran the same check. I broke one line in the home-screen banner code, the title of a slide. Plain pytest reported 8 failing tests. testmon selected 4. The other 4 broken tests never ran, and nothing in the output said they&rsquo;d been left out.</p>
<p>The fix is one line, <code>export COVERAGE_CORE=ctrace</code>, which puts coverage back on its classic C tracer. A different tracer means a different record, so I re-recorded: 12 minutes 3 seconds for 2,248 tests. Afterwards, the same kind of breaking edit ran all 11 tests it broke.</p>
<p>Update, 7 November: testmon 2.3.0, released on 2 November, now sets the C tracer itself, in process and for subprocesses, so a fresh install no longer needs the export. The check in the next paragraph still does.</p>
<p>That&rsquo;s the lesson I&rsquo;d pass on before anything else here. A test selector that selects too little looks exactly like success: fewer tests, a faster run, all green. So when you set one up, break something shared on purpose, run the plain suite and the selector, and check they fail the same tests. Then do it again whenever Python or coverage.py changes version.</p>

<h2 class="relative group">Where a coding agent comes in
    <div id="where-a-coding-agent-comes-in" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-a-coding-agent-comes-in" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I don&rsquo;t make every change in that repository myself any more. A coding agent, Claude Code in my case, makes a lot of the small ones, and a good agent runs the tests after each edit. Faced with an 11-minute suite, it has the same two bad options I had: wait, or guess which test file matters.</p>
<p>So the repository&rsquo;s agent instructions now say the wrapper script is the default way to run tests while working, with <code>--all</code> for a full run. The agent gets the same precise set of tests I do, after every edit, in seconds. It&rsquo;s also why step 6 gives the script its own database and cache: on a busy day there&rsquo;s more than one session running tests on the same machine, and a kept database must never meet somebody else&rsquo;s run.</p>
<p>None of this needs anything AI-specific from testmon. It&rsquo;s the same record and the same selection. It just matters more when whatever is making the changes makes a great many of them.</p>

<h2 class="relative group">Why CI got something coarser
    <div id="why-ci-got-something-coarser" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-ci-got-something-coarser" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>testmon has no CI mode. It behaves the same on a runner as on a laptop, and its author writes that sharing or caching <code>.testmondata</code> across CI runners is far less effective than their hosted service, testmon.net. A cached record would also be blind in all the ways above, and my CI runs the suite with Django&rsquo;s runner in four processes, so I left testmon out.</p>
<p>What CI got instead is the same idea at file level. The checks workflow now starts with a small job that diffs the push and decides which half of the repository it touched: the Python API, the mobile app, both or neither. Before this, CI ran both the Python and the app jobs on every push, docs-only ones included. Now a docs-only push runs nothing, and a push to the app alone skips the Django suite. I dry-ran the job against five real commits: the docs-only one ran nothing, the app-only one skipped the Python job, and the backend-only one skipped the app job. A cut-down example:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">changes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">outputs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">python</span><span class="p">:</span><span class="w"> </span><span class="l">${{ steps.diff.outputs.python }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">fetch-depth</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">diff</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">BEFORE</span><span class="p">:</span><span class="w"> </span><span class="l">${{ github.event.before }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">HEAD</span><span class="p">:</span><span class="w"> </span><span class="l">${{ github.sha }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          FILES=&#34;$(git diff --name-only &#34;$BEFORE&#34; &#34;$HEAD&#34;)&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          # Python&#39;s inputs include the non-Python files its tests read.
</span></span></span><span class="line"><span class="cl"><span class="sd">          PY=&#39;^(myapp/|myproject/|fixtures/shared\.json$|web/dist/|pyproject\.toml$|uv\.lock$)&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">          if printf &#39;%s\n&#39; &#34;$FILES&#34; | grep -qE &#34;$PY&#34;; then
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;python=true&#34; &gt;&gt; &#34;$GITHUB_OUTPUT&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          else
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;python=false&#34; &gt;&gt; &#34;$GITHUB_OUTPUT&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          fi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">python</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">needs</span><span class="p">:</span><span class="w"> </span><span class="l">changes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">needs.changes.outputs.python == &#39;true&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv run python manage.py test --parallel 4 --noinput</span></span></span></code></pre></div></div>
<p>The real one also runs everything when there&rsquo;s no range to compare: a new branch, a force push or a manual run.</p>
<p>The blind spot is the same one in a different form. testmon can&rsquo;t see a JSON file, and neither can a path filter unless you list it. So the pattern names the shared fixtures, the built bundle and the generated site explicitly, with a comment asking whoever edits it to keep the list honest. The file-level version is cruder, but it fails in the safe direction: if anything on the Python side changed, the whole suite runs.</p>

<h2 class="relative group">Is it worth it?
    <div id="is-it-worth-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#is-it-worth-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For a suite that takes minutes, yes. The real cost is what it can&rsquo;t see, and that&rsquo;s what the wrapper script is for. Even so, the laptop loop is where testmon belongs. CI still runs the whole suite whenever the Python side changes, and that&rsquo;s the run I trust before a deploy.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
