Until recently, every change to the web side of my marketplace app went out the same way as a change to the Django API: build the Docker image, push it, and roll it across every machine on Fly.io. The web front end (the mobile app’s web export, plus the marketing pages a static site generator builds) was copied into the image next to the API, so there was no other way to get it out. A CSS fix took about seven minutes and restarted eleven machines. One day in August I did that six times.
This post walks through what I replaced it with: a deploy script that builds both front-end trees, uploads them to object storage under a versioned prefix, flips one pointer object, waits until every machine is serving the new version, and purges Cloudflare’s cache through the API. On the other side, Django reads the pointer on every request and serves whichever version it names. The API doesn’t restart at all.
Why the old way was a problem#
The restart was the biggest cost. Eleven machines rolling means eleven cold boots, and on a shared CPU Django takes about thirteen seconds to come up, seven of which is just importing Django. I had a warm-up step at the end of the deploy script that hit every machine so a real visitor wouldn’t be the one to pay for that, which meant every web release ended with waiting on the slowest machine. I’ve written before about what those cold boots cost on Fly.io, in Scale to Zero, and Every Restart Is a 30-Second Outage. The short version is that every restart is a window where a visitor can get a 504, so a deploy path that restarts nothing closes that window for web releases entirely.
The second problem was that the static site’s output was split. The HTML went into the image, but the RSS feeds and the search index went straight to the bucket. They were built from the same source, just not at the same moment, so they could disagree with each other and nobody would notice until they did.
The third one is more about habit. When a release costs seven minutes and a restart, you stop doing small ones. I’d let three or four fixes pile up rather than ship them one at a time, and that’s exactly how a one-line change ends up going out next to something broken.
The layout in the bucket#
Everything lives in an S3-compatible bucket under a web/ prefix:
web/current ← a text file containing one version string
web/a1b2c3d-20260914021500/manifest.json ← every file in this version
web/a1b2c3d-20260914021500/app/... ← the app's web export
web/a1b2c3d-20260914021500/site/... ← the static site's output
web/9f8e7d6-20260912093012/... ← the previous version, still thereA version is the short git SHA plus a UTC timestamp, so you can read off what commit it is and when it went up. web/current is the only thing that changes when a release goes live.
What the deploy script does#
The whole thing is one script, run from the repo root. In order:
1. Refuse a dirty tree. The first thing it does is git status --porcelain, and if anything comes back it exits. Whatever is on disk is what gets uploaded, and the old deploy script for the marketing site once shipped uncommitted copy that way. There’s an override flag for when you know what you’re doing, but I haven’t used it. It then computes the version string from git rev-parse --short HEAD and the current time.
2. Build both trees. For the app it runs the web export with the production API and WebSocket URLs baked in, followed by a Brotli precompression pass. The precompression is worth explaining: Cloudflare will Brotli-compress on the fly, but at a low quality level, and on this bundle it measured worse than gzip (2.04MB against 1.66MB). Compressing once at build time with quality 11 gets it to 1.28MB, so the script writes a .br twin next to every JS, CSS and JSON file over 1KB. For the marketing site it runs the static site generator with minification on.
3. Upload. It walks both build directories and uploads every file to web/<version>/app/... and web/<version>/site/... with sixteen threads. Each object gets a Content-Type guessed from its filename (with the .br stripped first), and the Brotli twins get Content-Encoding: br as well. Only when every file is up does it write manifest.json, a JSON object with two lists of relative paths. The order matters: the Django side treats a version without a manifest as not existing, so a half-uploaded version can never be made current. Right now that’s a few hundred files and the upload takes about six seconds.
4. Flip the pointer. One put_object writes the version string to web/current. That’s the release.
5. Wait for the machines. Each API machine caches the pointer for five seconds, so for a moment after the flip some of them are still on the old version. The script polls a version endpoint on the API every half second and counts consecutive answers that carry the new version. Fly load-balances across all the machines, so a run of sixteen matching answers in a row (more than the machine count) is how it knows every one of them has re-read the pointer. One stale answer resets the count. It gives up after 90 seconds and tells you to check by hand; the first real flip took a bit over 30 seconds, which is why the limit isn’t lower.
6. Purge Cloudflare. One API call, covered below. If it fails, the script prints a warning and carries on.
7. Post to the team chat. The same webhook the runtime error alerts use gets a one-liner with the version and who ran it, so a burst of errors can be read against what just changed. Rollbacks post in amber.
Start to finish, a release is a bit over a minute, most of which is the web export.
The Django side#
Django was already the thing serving the front end. Every non-API URL is answered with the prerendered page from the web export for that route, and the marketing, privacy and support pages come from the static site’s output. Those views used to open files on disk. Now they read through one small layer that knows about the bucket.
Per request it resolves three things: the current version (from web/current, cached for five seconds), the manifest for that version (cached for as long as that version is current), and the file itself (a small in-memory LRU of file bodies, so a page’s HTML and its main bundle stay in memory). The manifest is what makes 404s cheap: if a path isn’t listed, the answer is no without asking the bucket. If the client accepts Brotli, it reads the .br twin instead and sets the encoding header. Assets go out with a one-year immutable cache header, since their filenames are content-hashed. Anything derived from the file list, such as the table that maps routes to prerendered pages, is cached per version too, so a new deploy rebuilds it on the first request and not before.
Then there’s the endpoint the deploy script polls:
def web_version(request):
# bundle: the layer that reads web/current and the manifest
response = JsonResponse({"source": bundle.source(), "version": bundle.version()})
response["Cache-Control"] = "no-store"
return responseIt answers with bucket:<version> in production or filesystem when it’s serving from the image. Each Fly machine answers for itself, so hitting it a dozen times tells you whether the fleet has converged. It’s also the quickest way to check what a machine thinks is live when something looks off.
Purging Cloudflare#
HTML on the site is cached at Cloudflare’s edge for five minutes. Without a purge, a deploy would be live on the machines but most visitors would keep getting the old page for up to five minutes, which defeats the purpose.
The purge is a single API call:
urllib.request.Request(
f"https://api.cloudflare.com/client/v4/zones/{zone}/purge_cache",
data=json.dumps({"hosts": ["example.com", "www.example.com"]}).encode(),
headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
)I purge by host rather than by URL so I don’t have to list every page. The assets don’t need purging at all: the filenames are content-hashed, so a new build points at new filenames and the old ones just stop being referenced. Only the HTML that references them needs to be refreshed.
The script waits for the machines before purging, on purpose. If it purged first, Cloudflare could refill its cache from a machine that was still on the old version, and that page would then sit at the edge for another five minutes.
If the purge fails, the script prints a warning and finishes anyway. The cache expires on its own within five minutes, so the worst case is a slow release rather than a broken one. I’d rather that than have the deploy abort at the last step.
Checking a build before it goes live#
A no-flip mode does everything up to step 3 and stops. The version is in the bucket but nothing serves it. To look at it, I set an environment variable on one machine that pins it to that build regardless of what web/current says. When it looks right, the rollback command makes it current. It’s called rollback but it’s really “point current at this version”, and going forward and going back are the same operation.
Rolling back#
Old versions stay in the bucket, so rolling back is the same command with an earlier version, and a list command shows what’s there, newest first. It does the same wait and purge as a forward deploy. I haven’t needed it yet. Nothing prunes old versions at the moment; they’re small and I’d rather have them than not.
What stayed the same#
The API still deploys the old way, through its own deploy script, and it should: it’s stateful and it’s where the migrations run. The Dockerfile still copies both front-end builds into the image too. That’s deliberate. When there’s no bucket configured (local development, the test suite) or the bucket can’t be read, the serving layer falls back to those directories on disk, so a storage outage means visitors get the front end that shipped with the last image rather than an error page. One environment variable forces that behaviour in production if the bucket ever needs to be taken out of the loop.
The front end changes far more often than the API does, and each release now takes about a minute instead of seven. I’ve stopped batching without trying to.