My Current Stack: Scaling Two Products on Minimal Compute

I’m currently shipping two products as a solo developer: a field estimation app for tradespeople who quote jobs on site, and a community marketplace platform with job listings, used goods, and real-time messaging.

Different products, different users, nearly identical stack — and both run entirely on shared-cpu-1x machines, the cheapest tier Fly.io offers. That constraint is deliberate. Instead of sizing servers for peak load, I squeeze as much as possible out of one small machine, measure exactly where it collapses, and let Fly’s autoscaler add and remove machines around that measured limit. This post covers the optimisations that make that possible, the load-test numbers behind the thresholds, and how scaling up and down actually works on Fly.io.

The Philosophy: Measure One Machine, Then Multiply

#

The scaling strategy for both apps is the same three steps:

1. Make a single small machine as fast as it can reasonably be.
2. Load-test it until it breaks, and note the concurrency level where latency collapses.
3. Configure Fly’s autoscaler to start a second machine just before that point, and stop idle machines when traffic drops.

Capacity then scales roughly linearly with machine count, and the baseline cost is one small VM per app. Everything below serves one of those three steps.

Making One Small Machine Fast

#

Granian with uvloop

#

Both backends are Django 6 + Django REST Framework, served by Granian, a Rust-based ASGI server, with uvloop as the event loop:

granian --interface asgi project.asgi:application \
    --workers 3 --loop uvloop --ws \
    --workers-kill-timeout 15 --respawn-failed-workers

Three workers is the sweet spot for a 1-vCPU shared machine — enough to overlap I/O waits, not enough to thrash the CPU. Granian speaks HTTP/1.1, HTTP/2, and WebSockets from one process, so there’s no separate proxy layer eating memory. Local Docker runs the same three workers, so my laptop approximates one production machine when testing.

Rust in the Hot Paths

#

Wherever a Rust-backed drop-in exists for a Python hot path, I take it:

• orjson via drf-orjson-renderer — every API response serialises through Rust instead of the stdlib JSON encoder.
• blurhash-rs — image placeholders computed at upload time.
• uuid-utils — Rust UUID generation.
• psycopg 3 (binary) — C-accelerated Postgres driver.

None of these change the architecture; they just raise the ceiling of what one vCPU can do before you need a second machine.

Kill the Hidden Latency Taxes

#

The single biggest win in either codebase wasn’t an algorithm — it was connection reuse. Under Granian’s thread pool, every new thread was building a fresh boto3 S3 client, and every endpoint that returns a photo or PDF paid that cost. A custom SharedConnectionS3Storage class caches the client per process instead:

Signed URLs went from ~150ms to ~1ms on new threads.

The same lesson applies to the database: Fly’s Postgres sits behind PgBouncer, so the app runs conn_max_age=0 and lets the pooler own connection reuse, with a 5-second connect_timeout to fail fast on stale SSL instead of hanging a worker.

Cache Where the Bottleneck Actually Is

#

Load testing the marketplace revealed something worth internalising: the ceiling was Postgres connections, not CPU. Browse traffic is overwhelmingly anonymous and identical across users, and under load those identical list queries saturate the connection pool while the CPU sits idle.

The fix is a Redis cache for anonymous list pages, with per-namespace version counters for invalidation — a write bumps one counter (O(1), no key bookkeeping) and every cached page for that resource is instantly stale. A short TTL backstops anything missed. Serving browse pages straight from Redis removes the real bottleneck and lets the small machine spend its connections on logged-in traffic.

Background Work Without a Broker

#

Both apps use django-tasks-db, Django’s database-backed task framework, instead of Celery. A separate db_worker process polls the database for jobs — image processing, PDF generation, machine translation, reminders. No broker, no extra service to size and monitor, and the API process never blocks on heavy work. The worker even connects straight to Postgres, bypassing PgBouncer, since long-running jobs don’t play well with transaction pooling.

Memory: jemalloc and a Swap Cushion

#

Image and PDF processing spike memory in ways steady-state sizing doesn’t predict. Two cheap mitigations: jemalloc preloaded in the Docker image to reduce fragmentation, and swap on every machine (2GB on the API machines, 1GB on the WebSocket machine) so a spike swaps to disk instead of getting the process OOM-killed. Swap is not a performance feature — it’s an insurance policy that lets me keep memory allocations small.

Load Testing: Finding the Knee

#

Thresholds are only defensible if they’re measured. Each project has its own harness:

• The estimation app has a custom Python stress harness (scripts/stress-test.py) that sweeps concurrency levels and reports RPS, p50/p95/p99, and error rates, plus management commands that generate hundreds of realistic jobs so list endpoints are queried against real data volumes.
• The marketplace uses Locust with a step-load shape that ramps 50 → 500 users in 30-second steps to find the saturation point, endpoint weights matching real browse-heavy traffic, and a separate k6 suite that ramps 5 → 50 virtual users against production with a p95 < 2s threshold.

Representative numbers from the estimation app’s sweep (3 Granian workers, Docker on my dev machine):

And the number that drives everything else, from load-testing the marketplace on a single shared-cpu-1x machine: p95 stays under 300ms up to ~5 concurrent requests, then collapses past ~8–10 — throughput falls and latency spikes to multi-second. That knee is the whole basis of the autoscaling config.

How Scaling Works on Fly.io

#

Fly’s model is simple: your app is a fleet of identical machines behind a proxy, and the proxy counts load per machine. You declare two numbers:

[http_service.concurrency]
  type = 'requests'
  soft_limit = 8    # past this, start another machine
  hard_limit = 20   # past this, shed load instead of queueing

auto_stop_machines = 'stop'
auto_start_machines = true
min_machines_running = 1

Scaling up: when in-flight requests on a machine exceed the soft limit, the proxy starts a stopped machine and routes new traffic there. Since the load test showed collapse at 8–10 concurrent requests, the soft limit sits at 8 — machine #2 starts right at the knee, not after users are already seeing multi-second latency. The hard limit of 20 is the circuit breaker: beyond it the proxy sheds load rather than letting one machine melt down.

Scaling down: when traffic drops, Fly stops idle machines automatically, down to min_machines_running = 1. A stopped machine costs nothing for compute (only its rootfs storage), and cold-starts in well under a second when the proxy needs it back. The baseline cost of each app is literally one small VM, with burst capacity always ready.

One subtlety: the concurrency type is requests, not connections. Modern clients hold keep-alive connections open, so counting connections barely moves under real load and the autoscaler never triggers. Counting in-flight requests tracks actual work.

WebSockets Scale on a Different Axis

#

The marketplace splits its WebSocket server into a second Fly app sharing the same Docker image. The API app runs 3 workers and counts requests (soft limit 8). The WebSocket app runs 1 worker on a 512MB machine and counts connections:

[http_service.concurrency]
  type = 'connections'
  soft_limit = 1500
  hard_limit = 2000

A long-lived socket is nearly free to hold open but occupies a slot for hours, while an HTTP request occupies a slot for milliseconds — the same autoscaling signal can’t serve both. Splitting them means each fleet scales on its own honest metric, and idle WebSocket capacity costs 512MB, not 2GB. On the Channels side, the Redis layer runs with capacity: 1500 and 60-second message expiry, plus socket keepalives every 30 seconds because Fly’s Redis proxy idle-closes the long-lived BRPOP sockets that Channels depends on.

Offloading Compute to the Edge and the Client

#

The cheapest server work is the work the server never does:

• Static and media files live on Tigris, Fly’s S3-compatible storage, served through its CDN with signed URLs for private media — zero image bytes transit the Django machines.
• AI inference runs on the user’s phone. Both apps embed a local LLM via llama.rn (the marketplace streams a quantised 4B model, ~2.7GB; the estimation app wires tool-calling into its data layer so the assistant can create and update jobs). Zero inference cost server-side, and it works offline.
• Offline-first mobile (Expo SDK 57, SQLite on device, Dexie on web, a sync queue with retries) means the apps batch writes and tolerate dead zones instead of hammering the API with chatty per-keystroke requests.

The Rest of the Stack, Briefly

#

Firebase Auth plus WebAuthn passkeys on both apps, with a custom middleware authenticating WebSocket connections before the upgrade. Postgres in production, SQLite in development. django-unfold for admin, django-simple-history for audit trails. uv and ruff for tooling. Mobile is React Native 0.86 / React 19 / TypeScript with Expo Router, Gluestack UI v3, and Tailwind 4, with OTA updates through self-hosted Expo Updates and subscriptions via RevenueCat. Quality gates: pytest + mypy + bandit on the backend, Jest + Detox + Playwright on the client.

What I’d Tell You to Steal

#

The transferable idea isn’t any single library — it’s the loop. Make one cheap machine genuinely fast (Rust hot paths, connection reuse, cache in front of the real bottleneck, offload what you can to CDN and client). Load-test until you find the exact concurrency where it breaks. Set the autoscaler’s soft limit at that knee, let idle machines stop, and keep the minimum fleet at one. Two production apps, real-time features, AI assistants — and the steady-state bill is a couple of the cheapest VMs on the platform.