Until recently, every change to the Maru web app went out the same way as a change to the Django API: build the Docker image, push it, and roll it across every machine on Fly.io. The web front end (the Expo web export, plus the marketing pages Hugo generates) was copied into the image next to the API, so there was no other way to get it out. A CSS fix took about seven minutes and restarted eleven machines. One day in August I did that six times.
This post walks through what I replaced it with: a deploy script that builds both front-end trees, uploads them to object storage under a versioned prefix, flips one pointer object, waits until every machine is serving the new version, and purges Cloudflare’s cache through the API. On the other side, a small Django module reads the pointer on every request and serves whichever version it names. The API doesn’t restart at all.
Why the old way was a problem#
The restart was the biggest cost. Eleven machines rolling means eleven cold boots, and on a shared CPU Django takes about thirteen seconds to come up, seven of which is just importing Django. I had a warm-up step at the end of the deploy script that hit every machine so a real visitor wouldn’t be the one to pay for that, which meant every web release ended with waiting on the slowest machine. I’ve written before about what those cold boots cost on Fly.io, in Scale to Zero, and Every Restart Is a 30-Second Outage. The short version is that every restart is a window where a visitor can get a 504, so a deploy path that restarts nothing closes that window for web releases entirely.
The second problem was that Hugo’s output was split. The HTML went into the image, but the RSS feeds and the search index went straight to the bucket. They were built from the same source, just not at the same moment, so they could disagree with each other and nobody would notice until they did.
The third one is more about habit. When a release costs seven minutes and a restart, you stop doing small ones. I’d let three or four fixes pile up rather than ship them one at a time, and that’s exactly how a one-line change ends up going out next to something broken.
The layout in the bucket#
Everything lives in the static bucket on Tigris under a web/ prefix:
web/current ← a text file containing one version string
web/a1b2c3d-20260914021500/manifest.json ← every file in this version
web/a1b2c3d-20260914021500/dist/... ← the Expo web export
web/a1b2c3d-20260914021500/hugo/... ← Hugo's output
web/9f8e7d6-20260912093012/... ← the previous version, still thereA version is the short git SHA plus a UTC timestamp, so you can read off what commit it is and when it went up. web/current is the only thing that changes when a release goes live.
What the deploy script does#
The whole thing is one script, scripts/deploy_web.py, run from the repo root. In order:
1. Refuse a dirty tree. The first thing it does is git status --porcelain, and if anything comes back it exits. Whatever is on disk is what gets uploaded, and the old Hugo deploy script once shipped uncommitted copy that way. There’s an --allow-dirty flag for when you know what you’re doing, but I haven’t used it. It then computes the version string from git rev-parse --short HEAD and the current time.
2. Build both trees. For the app it runs npm run build:web:prod, which is expo export --platform web with the production API and WebSocket URLs baked in, followed by a Brotli precompression pass and the admin block-editor bundle. The precompression is worth explaining: Cloudflare will Brotli-compress on the fly, but at a low quality level, and on this bundle it measured worse than gzip (2.04MB against 1.66MB). Compressing once at build time with quality 11 gets it to 1.28MB, so the script writes a .br twin next to every JS, CSS and JSON file over 1KB. For the marketing site it runs hugo --gc --minify in hugo-site/, which writes to public/.
3. Upload. It walks mobile-app/dist and public/, and uploads every file to web/<version>/dist/... and web/<version>/hugo/... with sixteen threads. Each object gets a Content-Type guessed from its filename (with the .br stripped first), and the Brotli twins get Content-Encoding: br as well. Only when every file is up does it write manifest.json, a JSON object with two lists of relative paths. The order matters: the Django side treats a version without a manifest as not existing, so a half-uploaded version can never be made current. Right now that’s 314 files and the upload takes about six seconds.
4. Flip the pointer. One put_object writes the version string to web/current. That’s the release.
5. Wait for the machines. Each API machine caches the pointer for five seconds, so for a moment after the flip some of them are still on the old version. The script polls /api/web-version/ every half second and counts consecutive answers that carry the new version. Fly load-balances across all the machines, so a run of sixteen matching answers in a row (more than the machine count) is how it knows every one of them has re-read the pointer. One stale answer resets the count. It gives up after 90 seconds and tells you to check by hand; the first real flip took a bit over 30 seconds, which is why the limit isn’t lower.
6. Purge Cloudflare. One API call, covered below. If it fails, the script prints a warning and carries on.
7. Post to Discord. The same webhook the runtime error alerts use gets a one-liner with the version and who ran it, so a burst of errors can be read against what just changed. Rollbacks post in amber.
Start to finish, a release is a bit over a minute, most of which is the Expo export.
The Django side#
Django was already the thing serving the front end. Every non-API URL is answered by a view that returns the prerendered page from the Expo export for that route, and the marketing, privacy and support pages come from Hugo’s output. Those views used to open files on disk. Now they go through one small module, web_bundle.py, which has a WebFiles class that every one of them calls instead:
body = web_files.read("dist", "tabs/home.html")Per request it resolves three things: the current version (from web/current, cached for five seconds), the manifest for that version (cached for as long as that version is current), and the file itself (an LRU of bodies, about 48MB per process, so a page’s HTML and its main bundle stay in memory). The manifest is what makes 404s cheap: if a path isn’t listed, the answer is no without asking the bucket. If the client accepts Brotli, it reads the .br twin instead and sets the encoding header. Assets go out with a one-year immutable cache header, since their filenames are content-hashed.
The same module builds the Expo route table, so /tabs/home maps to tabs/home.html, and it’s cached per version, so a new deploy rebuilds it on the first request and not before.
Then there’s the endpoint the deploy script polls:
def web_version(request):
response = JsonResponse({"source": web_files.source(), "version": web_files.version()})
response["Cache-Control"] = "no-store"
return response/api/web-version/ answers with bucket:<version> in production or filesystem when it’s serving from the image. Each Fly machine answers for itself, so hitting it a dozen times tells you whether the fleet has converged. It’s also the quickest way to check what a machine thinks is live when something looks off.
Purging Cloudflare#
HTML on marucommunity.com is cached at Cloudflare’s edge for five minutes. Without a purge, a deploy would be live on the machines but most visitors would keep getting the old page for up to five minutes, which defeats the purpose.
The purge is a single API call:
urllib.request.Request(
f"https://api.cloudflare.com/client/v4/zones/{zone}/purge_cache",
data=json.dumps({"hosts": ["marucommunity.com", "www.marucommunity.com"]}).encode(),
headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
)I purge by host rather than by URL so I don’t have to list every page. The assets don’t need purging at all: the filenames are content-hashed, so a new build points at new filenames and the old ones just stop being referenced. Only the HTML that references them needs to be refreshed.
The script waits for the machines before purging, on purpose. If it purged first, Cloudflare could refill its cache from a machine that was still on the old version, and that page would then sit at the edge for another five minutes.
If the purge fails, the script prints a warning and finishes anyway. The cache expires on its own within five minutes, so the worst case is a slow release rather than a broken one. I’d rather that than have the deploy abort at the last step.
Checking a build before it goes live#
--no-flip does everything up to step 3 and stops. The version is in the bucket but nothing serves it. To look at it, set WEB_BUNDLE_VERSION=<version> on one machine; that pins it to that build regardless of what web/current says. When it looks right, --rollback <version> makes it current. The flag is called rollback but it’s really “point current at this version”, and going forward and going back are the same operation.
Rolling back#
Old versions stay in the bucket, so rolling back is --rollback <version> with an earlier one, and --list shows what’s there, newest first. It does the same wait and purge as a forward deploy. I haven’t needed it yet. Nothing prunes old versions at the moment; they’re small and I’d rather have them than not.
What stayed the same#
The API still deploys the old way through deploy-all.sh, and it should: it’s stateful and it’s where the migrations run. The Dockerfile still copies mobile-app/dist and public/ into the image too. That’s deliberate. When there’s no bucket configured (local development, the test suite) or the bucket can’t be read, WebFiles falls back to those directories on disk, so a storage outage means visitors get the front end that shipped with the last image rather than an error page. WEB_BUNDLE_FROM_BUCKET=0 forces that behaviour in production if the bucket ever needs to be taken out of the loop.
The front end changes far more often than the API does, and each release now takes about a minute instead of seven. I’ve stopped batching without trying to.