<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Presigned URLs · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/presigned-urls/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Fri, 13 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/presigned-urls/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>How to Set Up AWS Emulator, the Free LocalStack Alternative</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/</guid>
      <pubDate>Fri, 13 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>AWS Emulator</category>
      <category>LocalStack</category>
      <category>Moto</category>
      <category>Floci</category>
      <category>S3</category>
      <category>Testing</category>
      <category>Docker</category>
      <category>GitHub Actions</category>
      <category>Python</category>
      <category>Presigned URLs</category>
      <category>OTA Updates</category>
      <description>Set up AWS&#39;s free emulator in Docker, on Linux and in GitHub Actions, point boto3 at it, and see what it caught in my S3 tests that LocalStack 4.12 missed.</description>
      <content:encoded><![CDATA[<p>AWS released <a href="https://github.com/aws/aws-emulator"  target="_blank" rel="noreferrer">AWS Emulator</a> on 11 November 2026. It&rsquo;s a local copy of the AWS APIs that you run on your own machine. It&rsquo;s free, it needs no AWS account, the code is public domain (CC0), and it covers more than 200 services. It arrived seven months after LocalStack started requiring an account and an auth token for its current image, so a lot of people are looking at it as the free LocalStack alternative they&rsquo;ve been waiting for.</p>
<p>This post is mostly the setup: Docker on a Mac or Windows machine, the Linux package, the environment variables that point the AWS CLI and boto3 at it, Docker Compose, pytest and GitHub Actions. I ran every command here myself, on version 1.0.342. After the setup comes what it got right and wrong when I pointed the S3 tests from two of my own projects at it, side by side with LocalStack 4.12, Moto and Floci. The two projects are the upload flow in my marketplace app, and the bucket a self-hosted Codemagic Patch server keeps OTA bundles in.</p>

<h2 class="relative group">AWS Emulator in one minute
    <div id="aws-emulator-in-one-minute" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#aws-emulator-in-one-minute" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>AWS doesn&rsquo;t publish a container image. You build one from a nine-line Dockerfile that installs their Linux package, run it on port 9297, and set a handful of environment variables. From then on your SDK talks to the emulator instead of AWS, and your code doesn&rsquo;t change:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">docker build -t aws-emulator .           <span class="c1"># the Dockerfile is below</span>
</span></span><span class="line"><span class="cl">mkdir -p ~/.aws/emulator
</span></span><span class="line"><span class="cl">docker run --rm -d --name aws-emulator -p 127.0.0.1:9297:9297 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">HOME</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">&#34;</span> -v <span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator:</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  aws-emulator -p <span class="m">9297</span> --telemetry off
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ENDPOINT_URL</span><span class="o">=</span>https://127.0.0.1:9297
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SHARED_CREDENTIALS_FILE</span><span class="o">=</span>~/.aws/emulator/9297/emulator_credentials
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CONFIG_FILE</span><span class="o">=</span>~/.aws/emulator/9297/config
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CA_BUNDLE</span><span class="o">=</span>~/.aws/emulator/9297/ca.pem
</span></span><span class="line"><span class="cl">aws sts get-caller-identity              <span class="c1"># &#34;Account&#34;: &#34;000000000000&#34;</span></span></span></code></pre></div></div>
<figure><video
    class="my-0 rounded-md"
    autoplay loop muted playsinline preload="metadata"
    poster="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-setup.jpg" style="background:url('data:image/webp;base64,UklGRloAAABXRUJQVlA4IE4AAACQBACdASogABIAPw1yskwmIyIhKAqoxiGJ6QDG9BPdK4I2hDwC1QaVpF521UAA/ui2Gzha/NBVWRJqzq2oTgehY/P2cqAeQAizoM8AAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="Terminal: the Dockerfile, starting the AWS Emulator container, a health check answering ok, aws sts get-caller-identity returning account 000000000000, an S3 upload, and a CORS preflight answered with HTTP 501">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-setup.mp4" type="video/mp4">
  </video><figcaption>The whole setup, recorded on my Mac: start, health check, the emulator&rsquo;s account, an S3 upload, and the one thing it can&rsquo;t do yet (a CORS preflight answers 501, more on that below).</figcaption>
</figure>

<p>It&rsquo;s ready in about half a second and idles at about 175 MiB of memory. Everything lives in memory, so stopping the container wipes it.</p>

<h2 class="relative group">Do you need an account? (LocalStack vs AWS Emulator)
    <div id="do-you-need-an-account-localstack-vs-aws-emulator" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#do-you-need-an-account-localstack-vs-aws-emulator" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>No. AWS Emulator, Moto and Floci all run without signing up for anything. AWS Emulator doesn&rsquo;t even want your AWS credentials: it makes its own access key every time it starts and only accepts requests signed with it.</p>
<p>LocalStack is the one that changed. Since March 2026 its current image needs a <code>LOCALSTACK_AUTH_TOKEN</code>, which comes from a free account (their free plan is aimed at hobby and non-commercial use) or a paid one. Without a token the container stops at startup. I tried both: the image tagged <code>latest</code> (2026.10.7, built 11 November) quits with exit code 55, while 4.12, the last community release and the version LocalStack itself suggests pinning, still starts without one.</p>
<figure><video
    class="my-0 rounded-md"
    autoplay loop muted playsinline preload="metadata"
    poster="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/localstack-needs-account.jpg" style="background:url('data:image/webp;base64,UklGRlYAAABXRUJQVlA4IEoAAABQBACdASogABIAPw10pUymIyGhKA1QxiGJ6QDJTPgAtfKr5WoM7/AUVx4AAP7ozNYDExNyKOZLAQjfzKlQvzdAcPxaBgAGLrjAAA==') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="Terminal: localstack/localstack:latest exits with &#39;License activation failed&#39; and exit code 55, then localstack/localstack:4.12 starts and its health check reports version 4.12.0 with S3 available">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/localstack-needs-account.mp4" type="video/mp4">
  </video><figcaption>LocalStack&rsquo;s current image without a token, then the pinned 4.12. Pinning works today, but 4.12 won&rsquo;t get fixes.</figcaption>
</figure>

<p>One thing AWS Emulator does want is telemetry, which is on by default. AWS says it never includes your requests, code or credentials, but if you&rsquo;d rather it sent nothing, there are three switches: <code>--telemetry off</code> on the command line, <code>EMULATOR_TELEMETRY=off</code> in the environment, or this in <code>~/.aws/emulator/config.toml</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">telemetry</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">enabled</span> <span class="p">=</span> <span class="kc">false</span></span></span></code></pre></div></div>

<h2 class="relative group">Set up AWS Emulator with Docker on macOS or Windows
    <div id="set-up-aws-emulator-with-docker-on-macos-or-windows" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#set-up-aws-emulator-with-docker-on-macos-or-windows" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Docker is the route on a Mac or a Windows machine. Put this in an empty folder as <code>Dockerfile</code>. It&rsquo;s AWS&rsquo;s own, from <a href="https://docs.aws.amazon.com/emulator/latest/userguide/getting-started-docker.html"  target="_blank" rel="noreferrer">their Docker guide</a>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="s">public.ecr.aws/amazonlinux/amazonlinux:2023-minimal</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">ADD</span> https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator-x86_64.rpm /tmp/aws-emulator.rpm<span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">RUN</span> rpm -i /tmp/aws-emulator.rpm <span class="o">&amp;&amp;</span> rm /tmp/aws-emulator.rpm <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="o">&amp;&amp;</span> mkdir -p /home/emulator <span class="o">&amp;&amp;</span> chown <span class="m">1000</span> /home/emulator<span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">USER</span><span class="w"> </span><span class="s">1000</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">ENV</span> <span class="nv">HOME</span><span class="o">=</span>/home/emulator<span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">EXPOSE</span><span class="w"> </span><span class="s">9297</span><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="k">ENTRYPOINT</span> <span class="p">[</span><span class="s2">&#34;aws-emulator&#34;</span><span class="p">,</span> <span class="s2">&#34;-b&#34;</span><span class="p">,</span> <span class="s2">&#34;0.0.0.0&#34;</span><span class="p">]</span></span></span></code></pre></div></div>
<p>On Apple silicon or any other ARM machine, change <code>x86_64.rpm</code> to <code>aarch64.rpm</code>. Then build it and start it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">docker build -t aws-emulator .
</span></span><span class="line"><span class="cl">mkdir -p ~/.aws/emulator
</span></span><span class="line"><span class="cl">docker run --rm -d --name aws-emulator --memory<span class="o">=</span>2g --memory-swap<span class="o">=</span>2g <span class="se">\
</span></span></span><span class="line"><span class="cl">  -p 127.0.0.1:9297:9297 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">HOME</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">&#34;</span> -v <span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator:</span><span class="nv">$HOME</span><span class="s2">/.aws/emulator&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  aws-emulator -p <span class="m">9297</span> --telemetry off</span></span></code></pre></div></div>
<p>The build took 88 seconds on my Mac and the image is 1.67 GB. Three parts of that <code>docker run</code> line matter more than they look:</p>
<ul>
<li><strong><code>-p 127.0.0.1:9297:9297</code>, not <code>-p 9297:9297</code>.</strong> The short form opens the emulator to your whole network.</li>
<li><strong>The volume and <code>-e HOME</code>.</strong> At startup the emulator writes four files into <code>~/.aws/emulator/9297/</code>: its credentials, an AWS <code>config</code> file, its certificate (<code>ca.pem</code>), and a Java trust store. Mounting the folder at the same path inside the container is how your SDK on the host gets to read them.</li>
<li><strong>The port appears three times.</strong> If you change it, change it everywhere. The emulator writes its port into those files, and two containers that share a folder will overwrite each other&rsquo;s.</li>
</ul>

<h2 class="relative group">Install AWS Emulator on Linux
    <div id="install-aws-emulator-on-linux" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#install-aws-emulator-on-linux" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On Linux you don&rsquo;t need Docker. There&rsquo;s a Debian package and an RPM:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Debian / Ubuntu (use aws-emulator_arm64.deb on ARM)</span>
</span></span><span class="line"><span class="cl">curl -fsSLo aws-emulator.deb https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
</span></span><span class="line"><span class="cl">sudo apt-get install -y ./aws-emulator.deb
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Amazon Linux, Fedora, RHEL (aarch64.rpm on ARM)</span>
</span></span><span class="line"><span class="cl">sudo dnf install -y https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator-x86_64.rpm
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">aws-emulator --telemetry off</span></span></code></pre></div></div>
<p>The package is 150 MB. It runs in the foreground on <code>127.0.0.1:9297</code> and writes the same four files to <code>~/.aws/emulator/9297/</code>. Ctrl+C stops it cleanly and deletes them. For test runs, add <code>--idle-timeout 900</code>, which shuts it down after 15 minutes without a request, so a forgotten emulator doesn&rsquo;t sit there all week.</p>

<h2 class="relative group">Point the AWS CLI and boto3 at the emulator
    <div id="point-the-aws-cli-and-boto3-at-the-emulator" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#point-the-aws-cli-and-boto3-at-the-emulator" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>These are the variables. Any SDK that reads the standard AWS environment picks them up, so the code under test needs no endpoint and no keys:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">E</span><span class="o">=</span>~/.aws/emulator/9297
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ENDPOINT_URL</span><span class="o">=</span>https://127.0.0.1:9297
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SHARED_CREDENTIALS_FILE</span><span class="o">=</span><span class="nv">$E</span>/emulator_credentials
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CONFIG_FILE</span><span class="o">=</span><span class="nv">$E</span>/config
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_CA_BUNDLE</span><span class="o">=</span><span class="nv">$E</span>/ca.pem
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_REGION</span><span class="o">=</span>us-east-1
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_EC2_METADATA_DISABLED</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_DISABLE_HOST_PREFIX_INJECTION</span><span class="o">=</span><span class="nb">true</span>
</span></span><span class="line"><span class="cl"><span class="nb">unset</span> AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">curl -s --cacert <span class="nv">$AWS_CA_BUNDLE</span> <span class="nv">$AWS_ENDPOINT_URL</span>/__admin/health   <span class="c1"># ok</span>
</span></span><span class="line"><span class="cl">aws sts get-caller-identity                                          <span class="c1"># 000000000000</span></span></span></code></pre></div></div>
<p>Then plain boto3 just works:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s2">&#34;s3&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">s3</span><span class="o">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="s2">&#34;uploads&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">s3</span><span class="o">.</span><span class="n">list_buckets</span><span class="p">()[</span><span class="s2">&#34;Buckets&#34;</span><span class="p">])</span></span></span></code></pre></div></div>
<p>The <code>unset</code> line is the one to keep. Environment variables beat the credentials file, so if your real AWS keys are still exported the SDK signs with them, the emulator refuses the request, and you lose ten minutes wondering why. <code>aws sts get-caller-identity</code> is a quick sanity check: the emulator is always account <code>000000000000</code>. If you see your own account number, you&rsquo;re talking to AWS.</p>
<p>Some other things I hit, or nearly did:</p>
<ul>
<li><strong>Keep <code>127.0.0.1</code> in the endpoint, not <code>localhost</code>.</strong> With an IP address the SDKs put the bucket name in the path. With <code>localhost</code>, the JavaScript, Java and Go SDKs move it into the host name (<code>uploads.localhost</code>), and the emulator doesn&rsquo;t accept that.</li>
<li><strong>It&rsquo;s HTTPS only, with a new certificate every start.</strong> boto3, the AWS CLI and Go read <code>AWS_CA_BUNDLE</code>. Node needs <code>NODE_EXTRA_CA_CERTS</code> pointing at the same <code>ca.pem</code>, and Java uses <code>truststore.p12</code> through <code>JAVA_TOOL_OPTIONS</code>. <code>AWS_CA_BUNDLE</code> replaces the trusted certificates for that process, so a test that also calls a real HTTPS API needs a combined bundle.</li>
<li><strong>There&rsquo;s no reset.</strong> No admin route empties it, so give every test its own bucket name, or restart the emulator between suites.</li>
<li><strong>The keys change on every start.</strong> Don&rsquo;t copy them into a <code>.env</code> file. Point at the credentials file.</li>
<li><strong>If your app builds its own client with an explicit endpoint</strong> (django-storages&rsquo; <code>AWS_S3_ENDPOINT_URL</code>, for example), set that from <code>AWS_ENDPOINT_URL</code> too, or that one client keeps talking to the real service.</li>
</ul>

<h2 class="relative group">Run AWS Emulator with Docker Compose
    <div id="run-aws-emulator-with-docker-compose" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#run-aws-emulator-with-docker-compose" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For a project that already has a <code>compose.yaml</code>, this service builds the image from the Dockerfile above, keeps the emulator&rsquo;s files in the project folder instead of your home directory, and has a health check, so <code>docker compose up --wait</code> returns once it&rsquo;s ready:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">aws</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">build</span><span class="p">:</span><span class="w"> </span><span class="l">./aws-emulator           </span><span class="w"> </span><span class="c"># the folder with the Dockerfile</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;-p&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;9297&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;--telemetry&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;off&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;127.0.0.1:9297:9297&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">HOME</span><span class="p">:</span><span class="w"> </span><span class="l">${PWD}/.aws-emulator-home</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">./.aws-emulator-home/.aws/emulator:${PWD}/.aws-emulator-home/.aws/emulator</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mem_limit</span><span class="p">:</span><span class="w"> </span><span class="l">2g</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">healthcheck</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">test</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;CMD&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;curl&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;-sf&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;--cacert&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;${PWD}/.aws-emulator-home/.aws/emulator/9297/ca.pem&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">             </span><span class="s2">&#34;https://127.0.0.1:9297/__admin/health&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">interval</span><span class="p">:</span><span class="w"> </span><span class="l">1s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">retries</span><span class="p">:</span><span class="w"> </span><span class="m">30</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">mkdir -p .aws-emulator-home/.aws/emulator
</span></span><span class="line"><span class="cl">docker compose up -d --wait aws       <span class="c1"># healthy in about 5 s with the image already built</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">E</span><span class="o">=</span><span class="nv">$PWD</span>/.aws-emulator-home/.aws/emulator/9297   <span class="c1"># then the variables from above</span></span></span></code></pre></div></div>
<p>Add <code>.aws-emulator-home/</code> to <code>.gitignore</code>. It holds that run&rsquo;s keys.</p>

<h2 class="relative group">Use AWS Emulator from pytest
    <div id="use-aws-emulator-from-pytest" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#use-aws-emulator-from-pytest" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>There&rsquo;s nothing emulator-specific in the tests themselves. What I add is a guard, because the failure I worry about isn&rsquo;t a red test. It&rsquo;s a green one that ran against a real bucket:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># conftest.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd"><span>@</span>pytest.fixture</span><span class="p">(</span><span class="n">scope</span><span class="o">=</span><span class="s2">&#34;session&#34;</span><span class="p">,</span> <span class="n">autouse</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">only_against_the_emulator</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;AWS_ENDPOINT_URL&#34;</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">url</span><span class="o">.</span><span class="n">startswith</span><span class="p">((</span><span class="s2">&#34;https://127.0.0.1&#34;</span><span class="p">,</span> <span class="s2">&#34;http://127.0.0.1&#34;</span><span class="p">)):</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;AWS_ENDPOINT_URL is </span><span class="si">{</span><span class="n">url</span><span class="si">!r}</span><span class="s2">; refusing to run against real AWS&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd"><span>@</span>pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">s3</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s2">&#34;s3&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd"><span>@</span>pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">bucket</span><span class="p">(</span><span class="n">s3</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">name</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;test-</span><span class="si">{</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span><span class="o">.</span><span class="n">hex</span><span class="p">[:</span><span class="mi">12</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>   <span class="c1"># no reset route, so never reuse a name</span>
</span></span><span class="line"><span class="cl">    <span class="n">s3</span><span class="o">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">name</span></span></span></code></pre></div></div>
<p>A test then reads like any other:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_presigned_upload_rejects_the_wrong_content_type</span><span class="p">(</span><span class="n">s3</span><span class="p">,</span> <span class="n">bucket</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;put_object&#34;</span><span class="p">,</span> <span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">300</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">Params</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Bucket&#34;</span><span class="p">:</span> <span class="n">bucket</span><span class="p">,</span> <span class="s2">&#34;Key&#34;</span><span class="p">:</span> <span class="s2">&#34;media/a.jpg&#34;</span><span class="p">,</span> <span class="s2">&#34;ContentType&#34;</span><span class="p">:</span> <span class="s2">&#34;image/jpeg&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="sa">b</span><span class="s2">&#34;&lt;html&gt;&#34;</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;text/html&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">                     <span class="n">verify</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;AWS_CA_BUNDLE&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">403</span></span></span></code></pre></div></div>
<p>That one is from my marketplace app&rsquo;s upload flow, and it&rsquo;s a good example of why the emulator you pick matters: three of the five I tried return 200 for it. More on that below. If you only run the affected tests on each change, <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-26-pytest-testmon/" >pytest-testmon</a> works with this unchanged.</p>

<h2 class="relative group">Run AWS Emulator in GitHub Actions
    <div id="run-aws-emulator-in-github-actions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#run-aws-emulator-in-github-actions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On a GitHub runner, skip Docker and install the Debian package. The background process from one step keeps running for the rest of the job:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">test</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">astral-sh/setup-uv@v10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Start AWS Emulator</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          curl -fsSLo aws-emulator.deb https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
</span></span></span><span class="line"><span class="cl"><span class="sd">          sudo apt-get install -y ./aws-emulator.deb
</span></span></span><span class="line"><span class="cl"><span class="sd">          aws-emulator --telemetry off --idle-timeout 900 &gt; emulator.log 2&gt;&amp;1 &amp;
</span></span></span><span class="line"><span class="cl"><span class="sd">          E=$HOME/.aws/emulator/9297
</span></span></span><span class="line"><span class="cl"><span class="sd">          for i in $(seq 30); do
</span></span></span><span class="line"><span class="cl"><span class="sd">            curl -sf --cacert $E/ca.pem https://127.0.0.1:9297/__admin/health &amp;&amp; break
</span></span></span><span class="line"><span class="cl"><span class="sd">            sleep 1
</span></span></span><span class="line"><span class="cl"><span class="sd">          done
</span></span></span><span class="line"><span class="cl"><span class="sd">          {
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_ENDPOINT_URL=https://127.0.0.1:9297&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_SHARED_CREDENTIALS_FILE=$E/emulator_credentials&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_CONFIG_FILE=$E/config&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_CA_BUNDLE=$E/ca.pem&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_REGION=us-east-1&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_EC2_METADATA_DISABLED=true&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            echo &#34;AWS_DISABLE_HOST_PREFIX_INJECTION=true&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          } &gt;&gt; &#34;$GITHUB_ENV&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">uv run pytest</span></span></span></code></pre></div></div>
<p>I ran the same steps in a fresh <code>ubuntu:24.04</code> container (with the arm64 package, since my Mac is ARM). The install took 29 seconds, nearly all of it the 150 MB download, and the health check answered within 4 seconds. Don&rsquo;t add real AWS credentials to this job at all. That&rsquo;s the cheapest way to be sure nothing in it reaches AWS.</p>
<p><code>LTS/latest</code> is whatever was released last, and as you&rsquo;ll see below that changes several times a day. If you want CI to be reproducible, AWS has a page on <a href="https://docs.aws.amazon.com/emulator/latest/userguide/versioning.html"  target="_blank" rel="noreferrer">installing a specific version</a>. For how I lay out the rest of a pipeline, see <a href="https://jared.lynskey.co.nz/en/posts/2026/2026-10-22-docker-compose-ci-github-actions/" >the GitHub Actions pipelines I run now</a>.</p>

<h2 class="relative group">Moving from LocalStack
    <div id="moving-from-localstack" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#moving-from-localstack" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If your tests already run against LocalStack, most of the move is configuration:</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>LocalStack</th>
					<th>AWS Emulator</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Endpoint</td>
					<td><code>http://localhost:4566</code></td>
					<td><code>https://127.0.0.1:9297</code> plus <code>AWS_CA_BUNDLE</code></td>
			</tr>
			<tr>
					<td>Credentials</td>
					<td>anything (<code>test</code> / <code>test</code>)</td>
					<td>the file it writes each run, or <code>--auth off</code></td>
			</tr>
			<tr>
					<td>Account ID</td>
					<td><code>000000000000</code></td>
					<td><code>000000000000</code></td>
			</tr>
			<tr>
					<td>Setup scripts</td>
					<td>init hooks in <code>/etc/localstack/init/ready.d</code></td>
					<td>none: create resources in fixtures, after the health check</td>
			</tr>
			<tr>
					<td>Image</td>
					<td><code>localstack/localstack</code> (token needed after 4.12)</td>
					<td>build your own from AWS&rsquo;s package</td>
			</tr>
	</tbody>
</table>
<p>The init hooks are the part that takes real work if you use them a lot. I didn&rsquo;t, because each test makes its own bucket. And nothing survives a restart: the emulator keeps everything in memory.</p>

<h2 class="relative group">What AWS Emulator caught in my S3 tests, and what it missed
    <div id="what-aws-emulator-caught-in-my-s3-tests-and-what-it-missed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-aws-emulator-caught-in-my-s3-tests-and-what-it-missed" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The setup is the easy part. What I wanted to know was whether it behaves like S3 where my code depends on S3&rsquo;s behaviour. I wrote 12 checks, each asserting what real S3 does according to AWS&rsquo;s documentation:</p>
<ul>
<li><strong>Six from the marketplace app&rsquo;s upload flow.</strong> The app hands the browser a presigned URL, the browser PUTs the photo straight into the bucket (so the bucket needs CORS), and an <code>ObjectCreated</code> event tells the backend the file has arrived. That&rsquo;s the flow from <a href="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/" >my post on object notifications</a>, with SQS standing in for the webhook.</li>
<li><strong>Six from a Codemagic Patch bucket.</strong> Patch is an OTA update server for React Native apps (<a href="https://jared.lynskey.co.nz/en/posts/2026/2026-09-13-moving-to-codemagic-patch/" >how I moved to it</a>). Its bucket is public, so phones can download bundles, except for an <code>_internal/</code> prefix where uploads wait to be published, and a bucket policy keeps that prefix private.</li>
</ul>
<p>I ran them against AWS Emulator 1.0.342 (once as it ships and once with <code>--auth off</code>), LocalStack 4.12 (as it ships, and with <code>S3_SKIP_SIGNATURE_VALIDATION=0</code>), Moto 5.2.6 in server mode, Floci 1.5.5, and Silo, the MinIO fork that Patch&rsquo;s self-hosted bundle runs.</p>
<figure><video
    class="my-0 rounded-md"
    autoplay loop muted playsinline preload="metadata"
    poster="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/s3-emulator-test-run.jpg" style="background:url('data:image/webp;base64,UklGRk4AAABXRUJQVlA4IEIAAAAQBACdASogABIAPw10rkymIyIhKAqoxiGJ6QDM0PgAtf/5Tl8J29EIQAD+6MzcUMCFo8DULSYQXf8FTN9rmZMAAAA=') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="Terminal: a script runs the same 12 S3 checks against seven emulator setups and prints a pass/fail matrix; every setup ends between 6 and 8 out of 12">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/s3-emulator-test-run.mp4" type="video/mp4">
  </video><figcaption>The real run. Nothing passed all 12.</figcaption>
</figure>

<p>No setup passed all 12; they all landed between 6 and 8. The totals matter less than which checks each one missed, so here are the ones that would have hurt me, worst first.</p>
<p><strong>AWS Emulator is the strictest about permissions.</strong> Take the presigned URL my app signs for one JPEG. Against LocalStack 4.12 as it ships, Moto and Floci, I could use that same URL to upload an HTML file and get a 200 back. On a bucket that serves public files, that&rsquo;s a stored-XSS problem waiting to happen. AWS Emulator refused it with a 403, as S3 does, and it refused the URL again once it had expired. (LocalStack 4.12 catches both if you set <code>S3_SKIP_SIGNATURE_VALIDATION=0</code>, and Floci catches the expiry but not the swapped file type.) It was also the only one that knew new S3 buckets have Block Public Access switched on, so a public bucket policy is refused until you turn that off, and the only one that applied an explicit <code>Deny</code> to the server&rsquo;s own signed requests as well as to strangers. These are exactly the mistakes that pass every test and then fail on the first real deploy.</p>
<p><strong>It can&rsquo;t test anything a browser does yet.</strong> A CORS preflight (the <code>OPTIONS</code> request a browser sends before a cross-origin PUT) gets <code>501 Not Implemented</code>. With authentication on, which is the default, it also refuses every unsigned request, so an anonymous download from a public bucket gets a 403 even when the policy allows it. Both gaps are already filed in the repository. The project keeps its known gaps as files under <code>issues/things-that-do-not-work/</code>, and on 13 November there were 6,840 of them, with 13,409 already moved to <code>things-that-now-work/</code>.</p>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="AWS Emulator&rsquo;s issue file for S3, titled &ldquo;7 net-new causes — CORS preflight answers 501…&rdquo;, in the things-that-do-not-work folder on GitHub"
    width="1400"
    height="804"
    src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_33033b091b284c0d.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_33033b091b284c0d.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_f0294a8a0a022038.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-cors-preflight-issue_hu_925176ba1af6cb05.webp"></span><figcaption>The CORS gap, filed by the project&rsquo;s own issue-finding agent on 9 November.</figcaption></figure>
</p>
<p><strong><code>--auth off</code> swaps one half for the other.</strong> With authentication off, anonymous reads work and the bucket policy is evaluated properly, so all six Patch checks pass. That&rsquo;s the only setup that managed it. But presigned URLs are no longer checked either, so the upload checks fall to two. Running two emulators on different ports, one with auth and one without, covers both halves, though it&rsquo;s one more thing to explain to whoever reads the CI file next.</p>
<p><strong>It&rsquo;s moving fast.</strong> Releases v1.0.290 to v1.0.342 came out in three days, each a long list of fixes (AWS says coding agents built it, and the pace shows), and one of the 11 November fixes was &ldquo;presigned GET refused with 403&rdquo; in S3. Anything in my table could be different by next week, in either direction. Pin a version in CI and move it deliberately.</p>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="The aws/aws-emulator releases page on GitHub: v1.0.342 marked Latest, released 3 hours earlier, with nine more releases listed in the sidebar"
    width="1600"
    height="1125"
    src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_e39a0eec6b783a7a.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_e39a0eec6b783a7a.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_df197ee02b7c435c.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/aws-emulator-releases_hu_59bd3abd9b9aaf92.webp"></span><figcaption>Ten releases in three days. Pin one.</figcaption></figure>
</p>

<h2 class="relative group">The Patch bucket: one policy condition that only matters on real S3
    <div id="the-patch-bucket-one-policy-condition-that-only-matters-on-real-s3" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-patch-bucket-one-policy-condition-that-only-matters-on-real-s3" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This was the part I didn&rsquo;t expect. Patch&rsquo;s self-hosted bundle keeps everything in one bucket, and the policy it applies to its bundled storage is the obvious one: anyone can read, except under <code>_internal/</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;Version&#34;</span><span class="p">:</span> <span class="s2">&#34;2012-10-17&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;Statement&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;Effect&#34;</span><span class="p">:</span> <span class="s2">&#34;Allow&#34;</span><span class="p">,</span> <span class="nt">&#34;Principal&#34;</span><span class="p">:</span> <span class="s2">&#34;*&#34;</span><span class="p">,</span> <span class="nt">&#34;Action&#34;</span><span class="p">:</span> <span class="s2">&#34;s3:GetObject&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;Resource&#34;</span><span class="p">:</span> <span class="s2">&#34;arn:aws:s3:::example-ota/*&#34;</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span> <span class="nt">&#34;Effect&#34;</span><span class="p">:</span> <span class="s2">&#34;Deny&#34;</span><span class="p">,</span> <span class="nt">&#34;Principal&#34;</span><span class="p">:</span> <span class="s2">&#34;*&#34;</span><span class="p">,</span> <span class="nt">&#34;Action&#34;</span><span class="p">:</span> <span class="s2">&#34;s3:GetObject&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;Resource&#34;</span><span class="p">:</span> <span class="s2">&#34;arn:aws:s3:::example-ota/_internal/*&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>On Silo, signed with the admin credentials, that does what it says: strangers can&rsquo;t read <code>_internal/</code>, and the server&rsquo;s own copy from <code>_internal/</code> into the public prefix (the &ldquo;publish&rdquo; step) goes through. On real S3 an explicit <code>Deny</code> with <code>&quot;Principal&quot;: &quot;*&quot;</code> applies to everyone, including your own credentials, so the same policy would block the server&rsquo;s own publish step. AWS Emulator refused that copy with a 403, which is what AWS&rsquo;s policy evaluation rules say S3 does.</p>
<p>Patch&rsquo;s maintainers clearly know this, because their <a href="https://github.com/codemagic-ci-cd/codemagic-patch/blob/main/deploy/selfhost/aws-s3-bucket-policy.example.json"  target="_blank" rel="noreferrer">example policy for AWS S3</a> adds one condition to the <code>Deny</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="s2">&#34;Condition&#34;</span><span class="err">:</span> <span class="p">{</span> <span class="nt">&#34;StringNotEquals&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;aws:PrincipalAccount&#34;</span><span class="p">:</span> <span class="s2">&#34;YOUR_AWS_ACCOUNT_ID&#34;</span> <span class="p">}</span> <span class="p">}</span></span></span></code></pre></div></div>
<p>With that line, AWS Emulator (with <code>--auth off</code>, so the anonymous half could be tested too) and Moto both let the worker publish while still refusing anonymous reads of <code>_internal/</code>. But only AWS Emulator caught the version without it. Moto, LocalStack 4.12 and Floci all let the copy through either way, so every one of them would have let me ship the broken policy. That&rsquo;s the clearest argument I found for testing policies against the strictest emulator available. Copy a policy from a MinIO setup to S3 without that condition and nothing complains until the first real publish.</p>
<p>(My own Patch server stores bundles on Tigris, which has no bucket policies at all, so I keep <code>_internal/</code> in a separate private bucket. That&rsquo;s what <code>S3_INTERNAL_BUCKET</code> is for in Patch&rsquo;s configuration. I also write paid guest posts about Patch for Codemagic. This isn&rsquo;t one of them, and nobody paid for it.)</p>

<h2 class="relative group">Which one I&rsquo;d use now
    <div id="which-one-id-use-now" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#which-one-id-use-now" aria-label="Anchor">#</a>
    </span>
    
</h2>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Ready in</th>
					<th>Idle memory</th>
					<th>Image</th>
					<th>Account</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>AWS Emulator 1.0.342</td>
					<td>0.4–0.5 s</td>
					<td>176 MiB</td>
					<td>1.67 GB (you build it)</td>
					<td>none</td>
			</tr>
			<tr>
					<td>LocalStack 4.12</td>
					<td>0.9–2.2 s</td>
					<td>90 MiB</td>
					<td>1.89 GB</td>
					<td>none for 4.12, token after</td>
			</tr>
			<tr>
					<td>Moto 5.2.6 (server)</td>
					<td>0.7–0.8 s</td>
					<td>45 MiB</td>
					<td>560 MB</td>
					<td>none</td>
			</tr>
			<tr>
					<td>Floci 1.5.5</td>
					<td>0.3–0.4 s</td>
					<td>11 MiB</td>
					<td>292 MB</td>
					<td>none</td>
			</tr>
	</tbody>
</table>
<p>Measured on my Mac in Docker, three cold starts each. None of them is slow enough to matter.</p>
<p>For backend tests I&rsquo;d start with AWS Emulator, because the mistakes it catches (permissions, signatures, Block Public Access) are the ones that otherwise only show up in production, and it&rsquo;s the only one in this list that&rsquo;s free, current and made by the people who make S3. Until the CORS preflight is implemented, I&rsquo;d keep one small job for the browser-facing checks (the preflight, and an anonymous download from the public prefix) on Floci. LocalStack 4.12 with strict signatures passes them too, but it&rsquo;s frozen and won&rsquo;t get fixes. If your tests are Python and you&rsquo;d rather not run a container, Moto&rsquo;s in-process mocks are still the lightest option. Just don&rsquo;t rely on it for presigned URL checks.</p>
<p>The 12 checks are short enough to write for your own buckets in an afternoon, and I&rsquo;d recommend doing that before you trust any emulator, this one included. Each tool in the table failed different checks, and those are the gaps your tests would quietly pass over.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/aws-emulator-setup/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>Upload Directly to S3 and Still Compress Images: Tigris Object Notifications with Django</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/</guid>
      <pubDate>Sat, 07 Nov 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Tigris</category>
      <category>S3</category>
      <category>S3 Event Notifications</category>
      <category>Presigned URLs</category>
      <category>Image Uploads</category>
      <category>Image Compression</category>
      <category>Object Storage</category>
      <category>Webhooks</category>
      <category>Django</category>
      <category>Fly.io</category>
      <category>Discord</category>
      <description>Direct-to-S3 uploads with presigned URLs are fast, but your backend never hears that the file arrived, so compression and thumbnails get awkward. Tigris object notifications fix that: the bucket POSTs to a Django webhook and the photo is processed while the user is still typing. The same signal turns log files into Discord alerts.</description>
      <content:encoded><![CDATA[<p>Every image upload I&rsquo;ve built has had the same tension in it. The person uploading wants the photo to go up fast. I want the server to compress it, make a thumbnail, pull a blurhash and a couple of colours out of it, drop the metadata that shouldn&rsquo;t be there, and store it the way the rest of the app expects. The easy way to get my list is to send the file through the API, and that is exactly what makes their list slow.</p>
<p>The fast way is a presigned URL. The API signs a URL, the client PUTs the file straight to the bucket, and the bytes never touch the API at all. It&rsquo;s the standard answer for good reason. But the moment you do it you&rsquo;ve cut your backend out of the loop. It handed out a URL and then heard nothing. Did the file arrive? Is it the size you expected? Is it even a JPEG? Something has to tell the backend, and the usual options are all a bit sad: trust the client to call back and say &ldquo;done&rdquo; (clients lie, crash and lose signal), poll the bucket on a schedule (slow and wasteful), or process lazily on the first read (the first viewer pays, and you still don&rsquo;t know when the file is safe to reference). This is what makes uploading directly to S3 look harder than it is. The upload is easy. The signalling is what&rsquo;s missing.</p>
<p>Object notifications are that signal. Tigris, the S3-compatible storage I use on Fly.io, will POST a small JSON body to your app every time an object is created or deleted in a bucket. On AWS the equivalent is S3 event notifications, which usually means wiring up a Lambda, an SQS queue or EventBridge before anything reaches your code (SNS can POST to an HTTPS endpoint, but then you&rsquo;re confirming subscriptions and verifying signatures first). Tigris keeps it to one HTTP request to a URL you own, which is about as simple as storage-to-backend signalling gets. The feature exists so the bucket can talk to your backend without you building a bridge for it.</p>
<p>This post is the whole flow: the setup, a Django endpoint that receives the events, a presigned upload that keeps every bit of server-side processing, and the gotchas (CORS, size limits, expiry, multiple files). At the end, the same signal doing back-office work, where log files landing in the bucket became Discord alerts and let me delete a cron job I&rsquo;d been nursing for months.</p>

<h2 class="relative group">The short answer
    <div id="the-short-answer" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-short-answer" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In the Tigris dashboard, open the bucket, then Settings, then Object notifications. Give it an HTTPS URL on your app and pick basic auth (a username and password Tigris sends in the <code>Authorization</code> header; a bearer token is the other option). Add a filter so you only hear about the keys you care about. The filter is a SQL-like expression, so &ldquo;only the three prefixes this post handles&rdquo; is:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">WHERE</span><span class="w"> </span><span class="o">`</span><span class="k">key</span><span class="o">`</span><span class="w"> </span><span class="n">REGEXP</span><span class="w"> </span><span class="s2">&#34;^(uploads/raw|logs/api|media)/&#34;</span></span></span></code></pre></div></div>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="Tigris bucket settings, Notifications tab: object notifications switched on, a webhook URL, basic authentication with a username and password, and a SQL-like key filter"
    width="1204"
    height="671"
    src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_7e9da7c6ca1125db.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_7e9da7c6ca1125db.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_4ddc968851d1f052.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-object-notifications-settings_hu_4ddc968851d1f052.webp"></span><figcaption>The whole setup: an HTTPS URL on your app, basic auth, and a filter on the key. The bucket name and URL here are swapped for examples.</figcaption></figure>
</p>
<p>Tigris then sends a batch like this on every change:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;events&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;eventName&#34;</span><span class="p">:</span> <span class="s2">&#34;OBJECT_CREATED_PUT&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;eventTime&#34;</span><span class="p">:</span> <span class="s2">&#34;2026-11-07T09:12:44Z&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;bucket&#34;</span><span class="p">:</span> <span class="s2">&#34;example-bucket&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;object&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;key&#34;</span><span class="p">:</span> <span class="s2">&#34;uploads/raw/1f3c.jpg&#34;</span><span class="p">,</span> <span class="nt">&#34;size&#34;</span><span class="p">:</span> <span class="mi">48213</span><span class="p">,</span> <span class="nt">&#34;eTag&#34;</span><span class="p">:</span> <span class="s2">&#34;9b2c…&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The event names you&rsquo;ll meet are <code>OBJECT_CREATED_PUT</code>, <code>OBJECT_CREATED_MULTIPART</code>, <code>OBJECT_CREATED_COPY</code> and <code>OBJECT_DELETED</code>, plus rename and storage-tier transitions. Your endpoint checks the auth, routes each event by key prefix, and returns a 2xx within ten seconds. Three things to plan for from the start: delivery is at least once, so you will see repeats; events are batched, so one request can carry several; and they can arrive out of order, so don&rsquo;t read &ldquo;latest&rdquo; off the order they came in.</p>

<h2 class="relative group">A Django webhook for the bucket&rsquo;s events
    <div id="a-django-webhook-for-the-buckets-events" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-django-webhook-for-the-buckets-events" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The view is boring on purpose. It checks the auth, parses the body and hands each event to a router:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">base64</span><span class="o">,</span> <span class="nn">hmac</span><span class="o">,</span> <span class="nn">json</span><span class="o">,</span> <span class="nn">logging</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">JsonResponse</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.views.decorators.csrf</span> <span class="kn">import</span> <span class="n">csrf_exempt</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.views.decorators.http</span> <span class="kn">import</span> <span class="n">require_POST</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">.bucket_events</span> <span class="kn">import</span> <span class="n">handle</span><span class="p">,</span> <span class="n">parse</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">_authorised</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">secret</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">header</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;Authorization&#34;</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">header</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;Basic &#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">password</span> <span class="o">=</span> <span class="n">base64</span><span class="o">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">header</span><span class="p">[</span><span class="mi">6</span><span class="p">:])</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="p">(</span><span class="ne">ValueError</span><span class="p">,</span> <span class="ne">UnicodeDecodeError</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">hmac</span><span class="o">.</span><span class="n">compare_digest</span><span class="p">(</span><span class="n">user</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span> <span class="sa">b</span><span class="s2">&#34;tigris&#34;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">hmac</span><span class="o">.</span><span class="n">compare_digest</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">password</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span> <span class="n">secret</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@csrf_exempt</span>
</span></span><span class="line"><span class="cl"><span class="nd">@require_POST</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">bucket_webhook</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">secret</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">BUCKET_WEBHOOK_SECRET</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">secret</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;not configured&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">_authorised</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">secret</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;unauthorised&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">401</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">body</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">body</span> <span class="ow">or</span> <span class="sa">b</span><span class="s2">&#34;</span><span class="si">{}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;invalid json&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">400</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">event</span> <span class="ow">in</span> <span class="n">parse</span><span class="p">(</span><span class="n">body</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">handle</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">logger</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">&#34;bucket event failed: </span><span class="si">%s</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span><span class="s2">&#34;ok&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span></span></span></code></pre></div></div>
<p><code>compare_digest</code> is there so the check takes the same time whatever you send it (on bytes, because on strings it raises for non-ASCII input, and a crafted header would get a 500 instead of a 401). If the secret isn&rsquo;t set, say on a fresh staging box, the endpoint refuses everything rather than accepting everything. And a handler that throws is logged and still answered 200: Tigris retries anything that isn&rsquo;t a 2xx, a retry would hit the same bug, and by then the event has been claimed (two sections down) so the retry would do nothing anyway. The log line is what gets it fixed. That assumes the failure is deterministic. A transient storage timeout consumes the event for good: the photos don&rsquo;t mind, because the save can process them itself, but a log file&rsquo;s alert would be lost. If that matters to you, release the claim in the <code>except</code> and answer 500 so Tigris retries, and accept that a real bug then retries until it&rsquo;s fixed.</p>

<h2 class="relative group">Routing by prefix
    <div id="routing-by-prefix" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#routing-by-prefix" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Everything in the bucket is already organised by prefix, so that&rsquo;s the natural way to route. Each use is one function and one line in a table:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">HANDLERS</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;uploads/raw/&#34;</span><span class="p">,</span>   <span class="n">on_raw_photo</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;logs/api/&#34;</span><span class="p">,</span>      <span class="n">on_api_log_file</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;media/&#34;</span><span class="p">,</span>         <span class="n">on_media_change</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">fn</span> <span class="ow">in</span> <span class="n">HANDLERS</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">prefix</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">claim</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>  <span class="c1"># see below</span>
</span></span><span class="line"><span class="cl">                <span class="n">fn</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span></span></span></code></pre></div></div>
<p>Adding a fourth thing later is a new function and a new row. I like that more than I expected to.</p>

<h2 class="relative group">At least once means &ldquo;maybe twice&rdquo;
    <div id="at-least-once-means-maybe-twice" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#at-least-once-means-maybe-twice" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Tigris retries if your endpoint is slow or returns an error, and like most webhook senders it makes no promise that you&rsquo;ll see an event only once. So before a handler runs, the event is claimed by <code>(key, eTag, eventName)</code>. The eTag matters: if a file is overwritten, it&rsquo;s a new version with a new eTag and should be handled again. A repeat of the same version shouldn&rsquo;t.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.core.cache</span> <span class="kn">import</span> <span class="n">cache</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">claim</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">ttl</span><span class="o">=</span><span class="mi">7</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">k</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;bucket-event:</span><span class="si">{</span><span class="n">event</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">event</span><span class="o">.</span><span class="n">etag</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">cache</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">ttl</span><span class="p">)</span>  <span class="c1"># False if someone already claimed it</span></span></span></code></pre></div></div>
<p><code>cache.add</code> only sets the key if it isn&rsquo;t there already, which in Redis is atomic, so two machines receiving the same retry can&rsquo;t both win. I use a database-backed lease for this in the real thing because I wanted it to survive a Redis flush, but the idea is identical.</p>

<h2 class="relative group">Direct-to-S3 image uploads, with the pipeline intact
    <div id="direct-to-s3-image-uploads-with-the-pipeline-intact" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#direct-to-s3-image-uploads-with-the-pipeline-intact" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is the part that pays for the plumbing.</p>
<p>Previously, picking photos for a listing meant the app held them, and when you hit Post it sent them all to the API as base64. The API resized them, made thumbnails, worked out colours and a blurhash, and only then replied. Ten photos on a slow connection was a long spinner. And a request carrying ten base64 photos is a big, slow body for an API server to hold in memory, which was quietly why I kept photos small.</p>
<p>Now it&rsquo;s four steps, and the backend is told about every one of them.</p>

<h3 class="relative group">1. The client shrinks the photo and asks for a slot
    <div id="1-the-client-shrinks-the-photo-and-asks-for-a-slot" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#1-the-client-shrinks-the-photo-and-asks-for-a-slot" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The app still resizes on the device, to the longest edge the server keeps (1080px) as a JPEG at 0.7 quality. That resize is for the user&rsquo;s bandwidth, nothing else. Then it asks the API for an upload slot:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">uuid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.utils</span> <span class="kn">import</span> <span class="n">timezone</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s2">&#34;s3&#34;</span><span class="p">,</span> <span class="n">endpoint_url</span><span class="o">=</span><span class="s2">&#34;https://fly.storage.tigris.dev&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">BUCKET</span> <span class="o">=</span> <span class="s2">&#34;example-bucket&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_upload_slot</span><span class="p">(</span><span class="n">user</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">public_id</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">key</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;uploads/raw/</span><span class="si">{</span><span class="n">public_id</span><span class="si">}</span><span class="s2">.jpg&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span> <span class="o">=</span> <span class="n">PhotoUpload</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">public_id</span><span class="o">=</span><span class="n">public_id</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">raw_key</span><span class="o">=</span><span class="n">key</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">expires_at</span><span class="o">=</span><span class="n">timezone</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">2</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;put_object&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">Params</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Bucket&#34;</span><span class="p">:</span> <span class="n">BUCKET</span><span class="p">,</span> <span class="s2">&#34;Key&#34;</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="s2">&#34;ContentType&#34;</span><span class="p">:</span> <span class="s2">&#34;image/jpeg&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">15</span> <span class="o">*</span> <span class="mi">60</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;upload_id&#34;</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;url&#34;</span><span class="p">:</span> <span class="n">url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;method&#34;</span><span class="p">:</span> <span class="s2">&#34;PUT&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;headers&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">:</span> <span class="s2">&#34;image/jpeg&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;expires_in&#34;</span><span class="p">:</span> <span class="mi">15</span> <span class="o">*</span> <span class="mi">60</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span></span></span></code></pre></div></div>
<p>The slot is a row before it&rsquo;s a file. That&rsquo;s what lets the server know who asked for it, when it expires, and later whether it was ever used.</p>

<h3 class="relative group">2. The client PUTs straight to the bucket
    <div id="2-the-client-puts-straight-to-the-bucket" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#2-the-client-puts-straight-to-the-bucket" aria-label="Anchor">#</a>
    </span>
    
</h3>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">slot</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">api</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="s2">&#34;/photo-uploads/&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">res</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">slot</span><span class="p">.</span><span class="nx">url</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">method</span>: <span class="kt">slot.method</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">headers</span>: <span class="kt">slot.headers</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">body</span>: <span class="kt">blob</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">res</span><span class="p">.</span><span class="nx">ok</span><span class="p">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">&#34;upload refused&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="c1">// keep slot.upload_id for the save
</span></span></span></code></pre></div></div>
<p>The API never sees the bytes. This is the same from a browser, from React Native or from a native app; the only difference is that a browser needs a CORS rule on the bucket, which is in the gotchas below.</p>

<h3 class="relative group">3. Tigris tells the backend, and the pipeline runs
    <div id="3-tigris-tells-the-backend-and-the-pipeline-runs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#3-tigris-tells-the-backend-and-the-pipeline-runs" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Within a few seconds Tigris posts an <code>OBJECT_CREATED_PUT</code> for <code>uploads/raw/…</code>, and the handler runs the same image pipeline the base64 path always used, while the person is still typing their title:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">BytesIO</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span><span class="p">,</span> <span class="n">ImageOps</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.core.files.base</span> <span class="kn">import</span> <span class="n">ContentFile</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.utils</span> <span class="kn">import</span> <span class="n">timezone</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">blurhash</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">MAX_BYTES</span> <span class="o">=</span> <span class="mi">15</span> <span class="o">*</span> <span class="mi">1024</span> <span class="o">*</span> <span class="mi">1024</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">on_raw_photo</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">event</span><span class="o">.</span><span class="n">name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;OBJECT_CREATED&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">event</span><span class="o">.</span><span class="n">size</span> <span class="o">&gt;</span> <span class="n">MAX_BYTES</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">s3</span><span class="o">.</span><span class="n">delete_object</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="n">Key</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span> <span class="o">=</span> <span class="n">PhotoUpload</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">raw_key</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">processed_at</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">upload</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="n">data</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">get_object</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="n">Key</span><span class="o">=</span><span class="n">event</span><span class="o">.</span><span class="n">key</span><span class="p">)[</span><span class="s2">&#34;Body&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">process</span><span class="p">(</span><span class="n">upload</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">to_webp</span><span class="p">(</span><span class="n">im</span><span class="p">,</span> <span class="n">longest</span><span class="p">,</span> <span class="n">quality</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">im</span> <span class="o">=</span> <span class="n">im</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">im</span><span class="o">.</span><span class="n">thumbnail</span><span class="p">((</span><span class="n">longest</span><span class="p">,</span> <span class="n">longest</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">buf</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">im</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="s2">&#34;WEBP&#34;</span><span class="p">,</span> <span class="n">quality</span><span class="o">=</span><span class="n">quality</span><span class="p">,</span> <span class="n">icc_profile</span><span class="o">=</span><span class="n">im</span><span class="o">.</span><span class="n">info</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;icc_profile&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">ContentFile</span><span class="p">(</span><span class="n">buf</span><span class="o">.</span><span class="n">getvalue</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">upload</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">Image</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">BytesIO</span><span class="p">(</span><span class="n">data</span><span class="p">))</span> <span class="k">as</span> <span class="n">im</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">im</span> <span class="o">=</span> <span class="n">ImageOps</span><span class="o">.</span><span class="n">exif_transpose</span><span class="p">(</span><span class="n">im</span><span class="p">)</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s2">&#34;RGB&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">im</span><span class="o">.</span><span class="n">thumbnail</span><span class="p">((</span><span class="mi">1080</span><span class="p">,</span> <span class="mi">1080</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">width</span><span class="p">,</span> <span class="n">upload</span><span class="o">.</span><span class="n">height</span> <span class="o">=</span> <span class="n">im</span><span class="o">.</span><span class="n">size</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">dominant_color</span> <span class="o">=</span> <span class="s2">&#34;#</span><span class="si">%02x%02x%02x</span><span class="s2">&#34;</span> <span class="o">%</span> <span class="n">im</span><span class="o">.</span><span class="n">resize</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span><span class="o">.</span><span class="n">getpixel</span><span class="p">((</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">image</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="si">}</span><span class="s2">.webp&#34;</span><span class="p">,</span> <span class="n">to_webp</span><span class="p">(</span><span class="n">im</span><span class="p">,</span> <span class="mi">1080</span><span class="p">,</span> <span class="mi">80</span><span class="p">),</span> <span class="n">save</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">thumbnail</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="si">}</span><span class="s2">_t.webp&#34;</span><span class="p">,</span> <span class="n">to_webp</span><span class="p">(</span><span class="n">im</span><span class="p">,</span> <span class="mi">320</span><span class="p">,</span> <span class="mi">70</span><span class="p">),</span> <span class="n">save</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">upload</span><span class="o">.</span><span class="n">blurhash</span> <span class="o">=</span> <span class="n">blurhash</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">im</span><span class="o">.</span><span class="n">resize</span><span class="p">((</span><span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">)),</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span><span class="o">.</span><span class="n">processed_at</span> <span class="o">=</span> <span class="n">timezone</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">upload</span><span class="o">.</span><span class="n">save</span><span class="p">()</span></span></span></code></pre></div></div>
<p>The first line of <code>on_raw_photo</code> matters more than it looks. The router sends every event under <code>uploads/raw/</code> here, including the deletes from the orphan sweep and from the size check itself, and <code>get_object</code> on a key that&rsquo;s gone would only raise.</p>
<p>So the client&rsquo;s shrink was only ever for the pipe. The server still decides the format (WebP), the sizes, the thumbnail, the placeholder, and the dimensions it stores. Re-encoding through Pillow also drops the EXIF block, which is where phone cameras put the GPS position, so the privacy strip comes free. The ICC profile would go the same way, which shifts wide-gamut phone photos a touch, so <code>to_webp</code> passes it through. If you&rsquo;d rather keep pixel-perfect originals, don&rsquo;t shrink on the client; the flow is identical, the PUTs are just bigger, and bigger PUTs are precisely the thing this design makes cheap. One catch: originals from an iPhone are HEIC, which Pillow only opens with <code>pillow-heif</code> installed.</p>

<h3 class="relative group">4. The save attaches the photos and never waits on the webhook
    <div id="4-the-save-attaches-the-photos-and-never-waits-on-the-webhook" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#4-the-save-attaches-the-photos-and-never-waits-on-the-webhook" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When they hit Post, the request carries <code>upload_ids</code> instead of image data. The server looks each one up, with three outcomes:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">take</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">upload_ids</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">uploads</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">PhotoUpload</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span> <span class="n">public_id__in</span><span class="o">=</span><span class="n">upload_ids</span><span class="p">,</span> <span class="n">expires_at__gt</span><span class="o">=</span><span class="n">timezone</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">uploads</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">upload_ids</span><span class="p">)):</span>     <span class="c1"># expired, or not this user&#39;s</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="n">UploadNotReadyError</span><span class="p">(</span><span class="n">upload_ids</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">upload</span> <span class="ow">in</span> <span class="n">uploads</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">upload</span><span class="o">.</span><span class="n">processed_at</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>          <span class="c1"># the webhook hasn&#39;t got here yet</span>
</span></span><span class="line"><span class="cl">            <span class="n">data</span> <span class="o">=</span> <span class="n">read_raw</span><span class="p">(</span><span class="n">upload</span><span class="p">)</span>              <span class="c1"># None if the PUT never landed</span>
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">raise</span> <span class="n">UploadNotReadyError</span><span class="p">(</span><span class="n">upload</span><span class="o">.</span><span class="n">public_id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="n">process</span><span class="p">(</span><span class="n">upload</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">uploads</span></span></span></code></pre></div></div>
<p>If the notification already ran, the processed files are attached as they are. If the file is in the bucket but the notification hasn&rsquo;t arrived, the save processes it inline, right there, so a post never waits on the webhook. The notification is a head start, not a dependency. And if the file isn&rsquo;t there at all, or an id has expired or belongs to someone else, the save returns a validation error the app retries after a moment; the app waits for its own PUTs before sending, so in practice that&rsquo;s rare.</p>
<p>A few details that made it hold up. The slot belongs to a user, so the server only accepts upload ids the same person created; otherwise anyone who saw an id could attach your photo to their listing. Everything falls back: if the slot request fails, the PUT is refused, or the API is an older version, the app sends base64 exactly as before. It only takes the upload path when every new photo has a slot, so a post&rsquo;s pictures are all uploads or all base64, and their order stays one list. And orphans expire: people pick photos and then give up, so an upload that&rsquo;s never attached expires after two days and the existing orphaned-media sweep cleans its files up.</p>

<h2 class="relative group">Presigned URL upload gotchas: CORS, size limits, expiry, multiple files
    <div id="presigned-url-upload-gotchas-cors-size-limits-expiry-multiple-files" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#presigned-url-upload-gotchas-cors-size-limits-expiry-multiple-files" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most of what goes wrong with presigned uploads goes wrong before the notification ever fires. These are the ones I&rsquo;d want on one page.</p>

<h3 class="relative group">CORS, for browsers only
    <div id="cors-for-browsers-only" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cors-for-browsers-only" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>A native app doesn&rsquo;t care. A browser sends a preflight <code>OPTIONS</code> to the bucket before the PUT, and the upload fails with a CORS error unless the bucket&rsquo;s CORS rules allow <code>PUT</code> from your origin with the <code>Content-Type</code> header. Tigris takes the standard S3 CORS configuration, so the rule looks the same as on AWS. It&rsquo;s the first thing to check when a presigned PUT works from <code>curl</code> and not from the page.</p>
<p><figure><span class="mc-img"><img
    class="my-0 rounded-md"
    loading="lazy"
    decoding="async"
    fetchpriority="auto"
    alt="Tigris bucket settings, CORS and headers tab: one rule allowing PUT from two origins with the Content-Type header"
    width="1204"
    height="671"
    src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_7f61261034a8a56.webp"
    srcset="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_7f61261034a8a56.webp 800w, https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_3aa47360232f6475.webp 1280w"
    sizes="(min-width: 768px) 50vw, 65vw"
    data-zoom-src="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/tigris-bucket-cors_hu_3aa47360232f6475.webp"></span><figcaption>One CORS rule is all a browser upload needs: your origins, PUT, and the Content-Type header the signature expects.</figcaption></figure>
</p>

<h3 class="relative group">The content type is part of the signature
    <div id="the-content-type-is-part-of-the-signature" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-content-type-is-part-of-the-signature" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Signing with <code>ContentType</code> means the client must send exactly that header, or the bucket answers 403 <code>SignatureDoesNotMatch</code>. That&rsquo;s why the slot returns the headers along with the URL. Sign it on purpose, but know what it buys you: the header is enforced, the content isn&rsquo;t, so a PDF labelled <code>image/jpeg</code> gets through. The pipeline is the real check; Pillow refuses to open what isn&rsquo;t an image, and the row is never marked processed.</p>

<h3 class="relative group">Size limits
    <div id="size-limits" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#size-limits" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>A presigned PUT URL carries no size limit. If you need a hard stop before the bytes land, a presigned POST with a <code>content-length-range</code> policy condition does that. I didn&rsquo;t bother, because the notification carries the object&rsquo;s <code>size</code>: the handler deletes anything over the limit before reading it, and the save-time path reads at most the limit plus one byte so it can tell. The app sends a few hundred kilobytes; anything near fifteen megabytes isn&rsquo;t ours.</p>

<h3 class="relative group">Expiry
    <div id="expiry" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#expiry" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Two clocks. The presigned URL is good for fifteen minutes, which is plenty for one PUT and short enough that a leaked URL is useless soon. The slot row is good for two days, which covers someone picking photos, putting the phone down and finishing the listing tomorrow. The client treats a pending upload as stale after a day and starts a fresh one rather than hoping.</p>

<h3 class="relative group">Multiple files, and bigger ones
    <div id="multiple-files-and-bigger-ones" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#multiple-files-and-bigger-ones" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>One slot per file. Ten photos is ten slots and ten parallel PUTs, each with its own progress (an <code>XMLHttpRequest</code> gives you <code>upload.onprogress</code>; <code>fetch</code> doesn&rsquo;t), which you can&rsquo;t get from one giant base64 POST. Past a few megabytes, use multipart (parts are 5 MiB minimum, so it&rsquo;s for files well beyond that): create the multipart upload on the server, presign each <code>upload_part</code>, and complete it when the parts are in. Tigris sends <code>OBJECT_CREATED_MULTIPART</code> when the upload is completed, so the same handler fires and nothing else changes.</p>

<h3 class="relative group">Out of order
    <div id="out-of-order" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#out-of-order" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Events can arrive out of order, because the object may have been written in more than one region. For uploads it doesn&rsquo;t matter: one key, one event. For a key that gets overwritten, trust the eTag and the object&rsquo;s <code>Last-Modified</code>, not the order the webhooks reached you.</p>

<h2 class="relative group">The same signal for logs: alerts to Discord
    <div id="the-same-signal-for-logs-alerts-to-discord" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-same-signal-for-logs-alerts-to-discord" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Once the bucket can talk to the backend, anything that lands in it can be reacted to the same way, and the second thing I pointed it at was logs.</p>
<p>The app&rsquo;s logs are shipped to the bucket as small gzipped files, one every five minutes or so per app. For most of the year a GitHub Action on a 15-minute cron read the last 20 minutes of that archive with DuckDB, found the slow-request lines, and posted a digest to Discord. On paper that&rsquo;s fine. In practice GitHub&rsquo;s scheduled workflows don&rsquo;t run on time when the platform is busy, so &ldquo;every 15 minutes&rdquo; was more like &ldquo;every few hours&rdquo;, and when it finally ran it had a bigger window to chew through and occasionally ran out of memory. The overlap I&rsquo;d added so nothing slipped between runs meant the same slow request sometimes showed up twice. It was a lot of machinery for &ldquo;tell me when a request is slow&rdquo;.</p>
<p>Now each log file is read once, as it lands. They&rsquo;re a few kilobytes gzipped. The handler pulls the slow-request lines out, and if there are enough of them, or one really slow one, it posts a digest to a Discord channel through a webhook. Same for a burst of 5xx responses in one file. A Discord webhook from Python is one call:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">discord_alert</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">settings</span><span class="o">.</span><span class="n">DISCORD_WEBHOOK_URL</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="n">text</span><span class="p">[:</span><span class="mi">1900</span><span class="p">]},</span>  <span class="c1"># Discord caps a message at 2000 chars</span>
</span></span><span class="line"><span class="cl">        <span class="n">timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span></span></span></code></pre></div></div>
<p>No cron, no overlap, no duplicates, and the alert arrives within a few minutes of the request instead of whenever GitHub got round to it. The Action is gone.</p>
<p>While each file is open anyway, I also count what&rsquo;s in it (errors, warnings, health checks) into an hourly tally table. The daily summary used to scan a whole day of the archive. Now it reads a few hundred rows, and only falls back to the archive if the tally has a gap.</p>

<h2 class="relative group">A guard on deletes
    <div id="a-guard-on-deletes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-guard-on-deletes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This one I didn&rsquo;t plan. It fell out of having <code>OBJECT_DELETED</code> events. When a media file is deleted, the handler checks whether any row still points at it. If one does, that&rsquo;s a bug somewhere (or a person in the bucket console with a heavy finger), so it clears the column (the app shows a placeholder instead of a broken image) and reports it.</p>
<p>It also counts deletions per hour. Normal housekeeping removes a handful. If it ever sees a couple of hundred in an hour, I want to hear about that while it&rsquo;s happening, not when users start reporting blank photos.</p>

<h2 class="relative group">Things worth knowing
    <div id="things-worth-knowing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#things-worth-knowing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Respond fast. Tigris gives each delivery ten seconds. Reading a 5 KB log file or processing one phone photo fits; anything that might take longer goes to a queue and the view returns 200, or Tigris times out and retries, and you&rsquo;re back to deduplicating.</p>
<p>Filter at the bucket. My log shipper writes a file every few minutes per app, forever. If you only care about one prefix, say so in the notification&rsquo;s filter so your app isn&rsquo;t woken for everything else.</p>
<p>Test with a fake body. The payload is small and stable, so the tests post JSON at the view with the right <code>Authorization</code> header and assert what the handler did. No real storage involved.</p>
<p>Don&rsquo;t make it load-bearing on day one. Every use here has a fallback: the save can process a photo itself, the daily summary can still scan the archive, the app can still send base64. Notifications made each of them faster and cheaper, but none of them falls over if a webhook goes missing.</p>
<p>The best bit of the week was deleting that workflow file. Forty lines of YAML, a Python script, two repository secrets and a recurring &ldquo;why didn&rsquo;t the alert fire&rdquo; are gone, and the replacement is one handler that reads one small file when it arrives. The uploads were the point, though. The photos go up as fast as the network allows, the server still gets the last word on every one of them, and nobody had to build a bridge between the bucket and the backend, because the bucket now just says what happened.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/tigris-object-notifications/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
