The DevOps Monitoring Tools I Use for Two Products, and the Ones I Dropped

For two small products, a marketplace and community app I build and an estimating app for tradespeople, both on Fly.io’s cheapest machines, my DevOps monitoring comes down to five things: ERROR logs posted to a chat channel, every log line archived in object storage and queried with DuckDB, Fly’s built-in Prometheus metrics, phone crash reports sent through Sentry’s open-source SDK to my own API, and one summary post each morning. There’s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here’s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.

What I mean by monitoring, and how small this is

#

By DevOps monitoring I mean finding out that something’s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.

Last year I wrote about setting up monitoring that developers actually use, mostly with teams in mind: a cloud provider’s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly’s shared-cpu-1x machines, as described in My Current Stack, and I’m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly’s metrics in September had the marketplace’s API averaging 1% CPU. This isn’t an enterprise stack, and I won’t dress it up as one.

Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy’s, on my workstation.

The list

#

Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly’s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)

Errors go where I already am

#

Until September, flyctl logs was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don’t alert.

The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can’t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand. The channel also carries what I want to read errors against: deploys post when they start, when they’re up and where they failed, with the commit and who ran it, and so do over-the-air updates and web releases. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.

In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process’s start command, and writes the same line to that process’s log, so an id I can’t find means a process that doesn’t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.

Logs: Fly keeps minutes, the archive keeps everything

#

Fly’s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly’s log shipper, which is Vector subscribed to the organisation’s log stream, with one change to its S3 sink:

[sinks.aws_s3]
  type = "aws_s3"
  bucket = "myapp-logs"
  compression = "gzip"
  key_prefix = "logs/{{fly.app.name}}/%F/"   # one folder per app per day
  batch.timeout_secs = 300                  # a quiet app still lands its lines

Every line from the marketplace’s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:

CREATE MACRO logs(app, day := '*') AS TABLE
  SELECT * FROM read_json_auto('s3://myapp-logs/logs/' || app || '/' || day || '/*.log.gz',
                               union_by_name := true);

SELECT timestamp, message FROM logs('myapp-api', '2026-09-06') WHERE message LIKE '% 500 %';

That macro is how I measured the cold starts in the scale-to-zero post: the startup script announces each phase, every line carries the machine’s id, and the rest is a GROUP BY.

So it’s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.

Metrics I didn’t have to build

#

Fly runs a managed Prometheus for each organisation, so I don’t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (fly_instance_cpu_balance matters on a shared CPU), responses and concurrency per app. Two things I wish I’d known first. It wants Authorization: FlyV1 <token>, and a Bearer header or a read-only token both get “something went wrong resolving organization”. And it keeps about twelve days.

It earned its place in September, when the logs had “Could not open a database connection” several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary’s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres’s start time never moved. Postgres hadn’t restarted; the host under it was stalling. I sent Fly’s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.

Crash reports from phones

#

A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry’s open-source SDK, but its DSN points at an endpoint in our own Django API that accepts Sentry’s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It’s errors only.

I didn’t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven’t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.

The only dashboard is a message

#

I haven’t built a dashboard for either app. Last year’s post had a test for them: if nobody opens one unprompted, it isn’t earning its place. A message that arrives on its own doesn’t need opening, so the closest thing I have is a post in the team chat every morning.

It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can’t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel. Then failed health checks, yesterday’s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what we’ve spent. A section that can’t load says so, and the post still goes out.

Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month’s Android totals with them taken out.

Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an “is active” checkbox, and I had no idea how many phones had taken one. The hosted service’s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up the move to a hosted update service separately.

What I dropped, or stopped believing

#

None of this was a paid tool. Mostly it was things I’d been treating as signals that weren’t, and one job running in the wrong place.

Cloudflare’s 504 rate, until I grouped it by user agent

#

In September the Cloudflare dashboard showed what looked like 22% of a day’s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app’s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in the scale-to-zero post as the cause.

The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare’s own Early Hints probes (bastion early hints and nginx-ssl early hints), which got no answer from the origin and were logged as 504s. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn’t doing anything for us anyway, since the origin sends no preload headers, so I turned it off.

The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user. So this is the rule I’d most like to pass on: before you believe a CDN’s error rate, group it by user agent and check it isn’t the CDN’s own machinery. On Cloudflare’s free plan that’s a GraphQL query on httpRequestsAdaptiveGroups, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn’t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.

A green health check

#

Fly checks /api/health/ on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it’s good at. But a health check runs against a machine that’s already up, so it can’t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now. Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain flyctl deploy leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.

The health check has also caused two outages of its own. In July, Django’s SECURE_SSL_REDIRECT answered Fly’s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an except Exception in the health view let a “connection already closed” error escape as a 500, and Fly pulled the machine in exactly the case the view’s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.

The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.

A scheduled job in CI

#

The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project’s service-account key, which can do anything in that project, and I didn’t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it’s past eight in Auckland and today’s post hasn’t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.

If I were starting again

#

I’d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That’s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I’d stop checking them.

I’d leave tracing until there’s a second service for a request to cross, and a hosted APM until there’s a team that needs shared dashboards, an on-call rota and a budget for someone else’s bill. That’s when the tools in last year’s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.