Scale to Zero, and Every Restart Is a 30-Second Outage

I went looking for a traffic spike on my Cloudflare dashboard and found something much worse underneath it: 22% of every request my app served in a day had failed. No alert had fired. Every health check was green. The logs looked perfectly healthy, because the failing requests never reached the application at all.

The cause turned out to be my own container startup script, doing sensible work at exactly the wrong time.

The spike was a red herring

#

The thing I set out to explain was one hour that took about 4,900 requests against a median of 121. Forty-two unique visitors. That is 110 requests per visitor, which is not how humans browse.

It was a scanner. A single bot walking through invented subdomains, asking each one for secrets:

api-stage-asia.marucommunity.com/.env.local
api-us-east-1-demo.marucommunity.com/.env
api-eu-uat.marucommunity.com/wp-config.php~
app-test-us-east-1.marucommunity.com/actuator/configprops
prod-api-use2.marucommunity.com/.env.local
relay.marucommunity.com/.env.production

Thirty or so hostnames, every .env variant you can think of, plus Spring Boot actuator endpoints on a Django app. All of them got a 301 to my canonical host and nothing was served. Boring, and genuinely harmless.

But while I had the analytics open, I grouped the same day by status code. That is where the real problem was.

Twenty-two percent. And not spread evenly across the site, but concentrated exactly where it hurts:

Those are the endpoints my mobile app polls every 30 seconds. So this was not an abstract error-rate number. It was every user’s phone, quietly failing to load their unread badge, all day, for weeks probably.

Why the logs looked fine

#

My first instinct was that the app had become slow. It had not. I pulled the application’s own request lines for one of the worst hours out of the log archive:

[2026-09-08 01:00:29] "GET /api/v1/notifications/unread_count/" 304 61.615
[2026-09-08 01:00:29] "GET /api/v1/notifications/"             304 63.568
[2026-09-08 01:00:59] "GET /api/v1/notifications/unread_count/" 304 58.742
[2026-09-08 01:00:59] "GET /api/v1/notifications/"             304 105.438

Forty-four to a hundred milliseconds, 304s all the way down. The origin was fast.

That is the important clue, and it is worth stating as a rule: if your edge reports errors your origin has never heard of, the requests are dying before they arrive. Stop reading application logs and start looking at what sits in front of them.

In my case that is Fly.io’s proxy, and the answer was in how my machines are scheduled.

Scale to zero is a promise about money, not latency

#

My config looks like this, and on the face of it it is reasonable:

[http_service]
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 1

Idle machines stop. Traffic starts them again. You pay for what you use. For a small app with bursty traffic this is exactly what you want, and it is why I had ten machines defined but almost always one running.

The bill it hides is latency. When a request arrives for a stopped machine, somebody has to wait for that machine to boot. If booting takes longer than the proxy’s patience, that somebody gets a 504.

So: how long did my machines take to boot? I had never measured. The log archive knows, because the startup script announces each phase, and every line carries the machine id:

WITH s AS (
  SELECT fly.app.instance AS inst, timestamp,
         CASE WHEN message ILIKE '%Starting Granian%'    THEN 'begin'
              WHEN message ILIKE '%Migrations complete%' THEN 'migrated'
              WHEN message LIKE  '%"GET /api/health/%'   THEN 'serving' END AS phase
  FROM logs('koreapost', '2026-09-08')
)
SELECT inst,
       min(timestamp) FILTER (WHERE phase = 'begin')    AS started,
       min(timestamp) FILTER (WHERE phase = 'migrated') AS migrated,
       min(timestamp) FILTER (WHERE phase = 'serving')  AS first_serve
FROM s GROUP BY inst

The answer, consistently, across every machine:

Then I counted how often that happened:

SELECT count(*) AS starts, count(DISTINCT fly.app.instance) AS machines
FROM logs('koreapost', '2026-09-08')
WHERE message ILIKE '%Starting Granian%'
-- starts: 92, machines: 10

Ninety-two cold starts in one day. Each one a half-minute window in which anything routed to that machine timed out. Twenty-two percent stopped looking mysterious.

The cause was in my own entrypoint

#

Here is what every single machine ran before it would answer a request:

if [ "$1" = "granian-api" ]; then
    echo "Running migrations..."
    python manage.py migrate --noinput

    echo "Running collectstatic in background..."
    python manage.py collectstatic --noinput &

    exec granian --interface wsgi koreapost_project.wsgi:application ...
fi

Read that again with “92 times a day” in mind.

migrate is a full Django boot: import every model, connect to Postgres, query the migrations table, decide there is nothing to do. On a shared CPU that is ten seconds to accomplish nothing. collectstatic is a second full Django boot that then uploads static files to object storage, competing for the same single core that the web server is trying to start on. Granian is the third boot.

Three Python interpreters, two of them doing work that had already been done, every time a machine woke up.

The reason it was written that way is good, and it is probably why yours looks similar: migrations have to run somewhere, Fly’s release_command had hung on me before, and the entrypoint is the one place guaranteed to execute on deploy. It is a correct place to put deploy work. It is just not a correct place to put it per machine start, and once your machines stop and start on their own, those are no longer the same event.

That is the whole bug, and I think it is a common one:

Work that belongs to a release was being run once per machine start. Scale-to-zero turned one event into ninety-two.

The fix: claim the release once

#

Migrations do not need to run on the eighth machine to boot a given image. They need to run once for that image, and everyone else needs to know it has happened.

That is a lock with a name that changes per deploy. Fly hands you the name for free in FLY_IMAGE_REF, and I already had Redis. The whole thing is a small script that runs before Django exists, so the common answer costs milliseconds rather than a framework boot:

def release_id() -> str:
    """Something that changes exactly when the deployed image does."""
    return os.getenv("FLY_IMAGE_REF") or os.getenv("FLY_MACHINE_VERSION") or ""


def claim() -> int:
    release = release_id()
    if not release:
        return PREPARE

    client = _client()
    if client is None:          # no Redis: behave exactly as before
        return PREPARE

    done_key, lock_key = _keys(release)

    if client.get(done_key):    # somebody already did it for this image
        return SKIP

    owner = os.getenv("FLY_MACHINE_ID", "unknown")
    if client.set(lock_key, owner, nx=True, ex=LOCK_TTL_SECONDS):
        return PREPARE          # we won the race; we do the work

    # Someone else is preparing. Wait for them, because serving requests
    # against a half-migrated schema is worse than starting slowly.
    deadline = _monotonic() + WAIT_SECONDS
    while _monotonic() < deadline:
        _sleep(POLL_SECONDS)
        if client.get(done_key):
            return SKIP
        if not client.get(lock_key) and client.set(lock_key, owner, nx=True, ex=LOCK_TTL_SECONDS):
            return PREPARE      # the owner died mid-migration; take over
    return PREPARE              # waited long enough; do it ourselves

The entrypoint becomes an if:

if python3 koreapost_project/release_gate.py claim; then
    python manage.py migrate --noinput
    python manage.py collectstatic --noinput &
    python3 koreapost_project/release_gate.py done
fi

exec granian --interface wsgi koreapost_project.wsgi:application ...

Every failure has to end in “prepare it anyway”

#

This is the part worth copying, more than the code.

A gate like this sits between your users and a schema migration. If it is wrong in the direction of skipping, you serve requests against a database that has not been migrated, which is a genuinely bad outage. If it is wrong in the direction of preparing, you run an idempotent no-op migration twice and waste ten seconds.

Those outcomes are not symmetrical, so the code must not treat them as such. Every uncertain path returns PREPARE:

• No Redis, or Redis unreachable. Prepare. This is exactly the old behaviour, so a cache outage degrades to “slow”, not “broken”.
• No image reference in the environment. Prepare. I do not know what release this is, so I cannot know whether it is ready.
• The lock’s owner died mid-migration. Its lock expires, the next waiter takes it over and prepares.
• Waited longer than the timeout. Prepare anyway. A machine that never serves is worse than a duplicated migration.

There is one deliberate asymmetry in the other direction: a machine that finds another machine actively holding the lock waits rather than serving. Starting slowly during a deploy is fine. Answering queries against a half-applied schema is not.

Watch out for one trap if you build this with a resilient cache wrapper, as I had. Mine returns False from add() when Redis is unreachable, which reads identically to “someone else holds the lock” and would have made every machine skip its migrations. That is precisely the dangerous direction. Distinguish the two by reading the lock back: a lock nobody holds means the cache is broken, not that you lost a race.

Testing it without a Redis

#

The logic is worth unit testing, because the interesting paths are the ones you cannot reproduce by hand. A fake with two methods covers it:

class FakeRedis:
    """Enough of redis-py for the gate: get, and set with nx/ex."""

    def __init__(self) -> None:
        self.store: dict[str, str] = {}

    def get(self, key): return self.store.get(key)

    def set(self, key, value, nx=False, ex=None):
        if nx and key in self.store:
            return None
        self.store[key] = value
        return True

Then drive the awkward cases by making time pass on your terms. Note that the gate calls module-level _sleep and _monotonic aliases rather than time.sleep directly, so a test can replace them without patching the time module for every other thread in the process:

def test_a_waiter_takes_over_when_the_owner_disappears(self):
    self.assertEqual(release_gate.claim(), release_gate.PREPARE)   # owner takes the lock
    _done, lock_key = release_gate._keys(release_gate.release_id())

    def owner_dies(_seconds):
        self.redis.store.pop(lock_key, None)                       # its lock expired

    with patch.object(release_gate, "_sleep", owner_dies):
        self.assertEqual(release_gate.claim(), release_gate.PREPARE)

Did it work? Partly

#

Deploy, stop a machine, start it again, watch the clock. The startup now announces its decision and gets out of the way:

15:23:34  Starting Granian (API server — HTTP/1.1 + HTTP/2)...
15:23:36  release gate: this release is already prepared, starting straight away
15:23:44  "GET /api/health/ HTTP/1.1" 200

Ten seconds, and a second machine measured eleven. Against 25 to 33 before, two thirds of the window is gone, and the gate itself costs two seconds because it never imports Django.

But ten seconds is still ten seconds. A request landing in the first of them still fails. So I went looking for the rest.

The bytecode nobody was caching

#

The image sets a line that appears in more or less every Python Dockerfile ever written:

ENV PYTHONDONTWRITEBYTECODE=1

It is good advice. A container should not be scribbling .pyc files into a layer at runtime. What I had never thought through is the other half of the bargain: if the runtime never writes bytecode, and the build never writes it either, then nothing is ever cached, and every process start compiles the entire dependency tree from source.

I counted:

site-packages .py  files : 5,707
site-packages .pyc files : 0

Django, DRF, every library, recompiled on each of those 92 daily starts. Measured on a production machine:

The fix is one line, and compileall writes explicitly so it works despite the env var:

RUN python -m compileall -q /app/.venv/lib /app/koreapost_project /app/marketplace /app/utils || true

Sixteen seconds of build time, once, in a cached layer.

While I was there I found something quieter. .dockerignore said:

__pycache__/
*.pyc

Docker matches those against the whole relative path, not each path component, so an unanchored pattern only ever matches at the root of the build context. Every nested __pycache__ was shipping. The image contained 218 .pyc files compiled by Python 3.15 on a laptop, riding along inside an image whose interpreter is 3.14 and quietly ignoring all of them. The patterns want to be **/__pycache__/ and **/*.pyc.

Honest result: end to end this bought about a second, not two. Ten seconds became nine. The isolated benchmark oversold it, as isolated benchmarks do.

Suspend, and the one-line setting that was blocking it

#

At nine seconds I had run out of work to remove. What was left was one unavoidable boot: Python starting, Django importing, the WSGI app coming up on a shared core.

So stop booting. Fly can snapshot a machine’s memory instead of shutting it down:

[http_service]
  auto_stop_machines = 'suspend'   # was 'stop'

I went to test it and Fly refused, with the most useful error message of the whole exercise:

failed to suspend VM: failed_precondition:
Machines with swap cannot be suspended

My config had this near the top, with a comment I had written myself and believed:

# Swap cushion so a memory spike swaps to disk instead of getting OOM-killed.
swap_size_mb = 2048

A reasonable precaution. Was it doing anything? Every machine said:

MemTotal:   985220 kB
SwapTotal: 2097148 kB
SwapFree:  2097148 kB

Not one page had ever been swapped. And the day’s logs held no OOM kills — the eleven lines that matched “oom” turned out to be requests for a JavaScript bundle whose content hash happens to contain those three letters. Meanwhile the actual memory guard was somewhere else entirely, in the server command: --workers-max-rss 800 respawns a worker long before it can reach the 1GB ceiling.

So the cushion was insurance against an event that had never happened, already covered by a different mechanism, and its premium was the ability to suspend. Out it went.

Three trials, all within half a second. Every endpoint answered 200 in 250 to 435 milliseconds immediately after waking.

What happens to your sockets while the machine sleeps

#

Nothing good, and there is no hook to fix it. Fly freezes the VM; the process is not signalled and cannot close anything on the way down. Whatever connections it held are still in its memory when it wakes, pointing at sockets the other end abandoned minutes ago.

You do not solve that going down. You solve it coming up, and mostly by having already made the right decisions:

• Database connections are opened per request (conn_max_age = 0), so there is nothing long-lived to go stale.
• The Redis cache degrades instead of raising. A wrapper catches connection errors and returns a miss.

I got to watch the second one earn its place. One resume logged exactly the failure you would expect:

redis.exceptions.ConnectionError: Error while reading from fly-...

and the request it happened inside returned 200 in 3.6 seconds, because a dead cache is a slow page, not an error page. Later resumes logged nothing at all.

If your cache client raises on a dead connection, suspend will convert every wake into a burst of 500s. Check that before you flip the setting, not after.

The test that lied to me

#

My first suspend measurement said the feature was a disaster: 502, after 30 seconds, twice.

I had pinned the request to a specific machine with fly-force-instance-id, which is a lovely header for load testing one instance and a terrible one for this. Forcing an instance bypasses the proxy logic that would have woken it. Fly told me so, if I had read it as an answer rather than an error:

machine was recently stopped and is unavailable to service request

The number that matters comes from the path real traffic takes. Forty concurrent requests through the edge, well past the soft limit of 12, so the proxy has to scale out onto suspended machines:

status codes:  40 × 200
slowest:       2.1s

Two lessons, and the second one is the expensive one. Measure the path your users take, not the path that is convenient to instrument. And when your tooling refuses to do something, read the refusal: “Machines with swap cannot be suspended” was the sentence that unlocked this entire afternoon.

What I would check in your app

#

If you run anything on a platform that stops idle instances, these questions took me an afternoon and would have saved me weeks of quiet failure:

1. How long does a cold start take? Not how long the container takes to run, how long until it answers a real request. If you cannot answer in seconds, measure it before you need to.
2. What does your entrypoint do per start that belongs to a deploy? Migrations, static file collection, cache warming, index building. All fine once. All expensive ninety-two times.
3. Is anything caching your bytecode? PYTHONDONTWRITEBYTECODE with no compileall in the build means the answer is no.
4. Can you suspend instead of stop? And if not, what is stopping you? Mine was a swap file that had never been touched.
5. Does your edge see errors your origin does not? Compare the two. A gap between them is not a reporting quirk, it is requests dying in the space between, and your application logs will never show it.

The health checks were green the whole time, incidentally. They always are: a health check runs against a machine that is already up. It cannot report on the thirty seconds before it existed.