There’s a gap between “works on my machine” and “works in production”, and monitoring is supposed to bridge it. Most of the time it doesn’t, because it gets built as an ops tool rather than something a developer reaches for when things break.
I’ve just gone freelance doing DevOps and infrastructure work, and the monitoring setup is one of the first things I look at anywhere I land — it tells you more about a team than the README does. The failure mode is nearly always the same: the tools exist, but they don’t connect to how developers actually work. Someone burns three hours on a bug that production logs would’ve solved in five minutes. Failures arrive as support tickets instead of alerts. Or the opposite: 200 alerts a day, most of them noise, so everyone ignores all of them, and the first half hour of every incident is spent working out what broke rather than fixing it.
Metrics, logs, and traces — linked, or useless#
Metrics tell you something is wrong (error rate spiked, latency jumped). Logs tell you what happened (stack traces, request payloads, error messages). Traces tell you where (which service, which endpoint, which database call). None of that is news. What matters is whether they’re linked: when an alert fires, you should be able to click from the metric to the relevant logs to the trace. If developers are manually correlating timestamps across three tools, the setup has failed, no matter how good each tool looks on paper.
Dashboards deserve the same test. CPU, memory and disk I/O matter for capacity planning, but they don’t help anyone debug a 500. The dashboards people open voluntarily show business metrics next to technical ones (signups per hour beside error rate), recent deployments overlaid on the timeline, the top five errors in the last hour with links to logs, and latency percentiles — p50, p95, p99 — rather than averages. If nobody opens a dashboard unprompted, it isn’t earning its place.
Alerts, same idea. Every alert should answer two questions: what’s broken, and where do I start looking?
Bad alert: “High CPU on web-server-3”
Good alert: “Error rate > 5% on /api/payments since 14:32. Last deploy: 14:15 by @sarah. [View logs] [View trace]”
A few things push alerts toward the second kind: alert on deviation from a baseline rather than an arbitrary threshold, route by ownership (payment errors go to the payments team, not the whole org), and group related errors into one Slack message instead of 50. Keep the loop tight, too — the time between “something broke” and “a developer knows” should be under a minute, which means real-time log streaming rather than five-minute batches, deploy markers on every dashboard, and tracing that survives service boundaries.
Build it for the people who’ll use it#
The biggest mistake ops teams make is building monitoring for themselves. The fix isn’t sophisticated. Sit with a developer during a debugging session and watch where they get stuck. Ask what questions come up mid-incident — “which service?”, “what changed?”, “what did the request look like?”. Find out which metrics actually matter for the product; it’s rarely CPU, and usually things like checkout completion rate or upload success rate.
Then remove friction. If instrumenting a new service takes a day, it won’t happen. Provide libraries that auto-instrument the frameworks you already use (Django, Express, Spring), copy-paste templates for dashboards and alerts, and self-service metrics so nobody has to file a ticket. And treat the whole thing as code: config in version control, alert rules tested so you know they fire when they should, and multi-region considered early if that’s where you’re headed.
Small team on AWS? Start with CloudWatch#
If you’re under 50 engineers and on AWS, CloudWatch is the right starting point, and I’d resist buying anything fancier on day one. EC2, Lambda, RDS and ECS all report to it automatically, so you get visibility without writing instrumentation. It runs $10–50 a month for a small team with no platform fee, and metrics, logs, traces (via X-Ray) and alarms all live in one place. You can have meaningful alerts and dashboards up in an afternoon, not weeks.
CloudWatch Logs Insights is the underrated bit — you can query logs without standing up Elasticsearch:
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)Beyond that:
- Composite alarms cut noise — alert when error rate is high and response time is degraded, not on each condition separately.
- Custom metrics are where the real value is. Push business metrics (signups, transactions, feature usage) through the CloudWatch SDK alongside the infrastructure ones.
- CloudWatch Synthetics runs scheduled canaries through real user journeys, so you learn the checkout is broken before your users do.
- X-Ray gives you distributed tracing with minimal setup — good enough for most microservice architectures.
You’ll know when you’ve outgrown it: you go multi-cloud, you want proper anomaly detection, dashboard customisation starts to hurt, or the team passes 50 engineers and needs real collaboration features.
Bigger teams: DataDog#
DataDog is expensive — $20–100K+ a year for a large org is normal — but at scale it earns its keep. One view across AWS, Azure, GCP, on-prem, containers, serverless, databases and frontend. Watchdog flags anomalies without hand-tuned thresholds, which matters because nobody can manually watch thousands of services. And you get the collaboration layer CloudWatch lacks: team dashboards, RBAC, shared investigation notebooks, PagerDuty/Opsgenie integration, multi-condition alerts, threshold forecasting, maintenance windows, and an APM that goes down to code-level profiling with auto-generated service maps.
Rolling it out, in roughly the order I’d do it: instrument critical services first and tag everything by team and environment from day one. Agree naming conventions, dashboard templates, severity levels and SLO definitions early — retrofitting standards is miserable. Wire it into CI/CD deploy markers, incident management and Slack. Budget time for training, because DataDog is powerful but not self-explanatory. And watch the bill: filter noisy metrics, sample traces on high-volume services, and audit which features you’re actually paying for.
Worth a look before committing: New Relic (similar, sometimes cheaper for high-volume tracing), Dynatrace (strong AIOps, popular in financial services), Splunk (best-in-class log analysis, especially if security already runs it), and Grafana Cloud (the natural home if you’re already on Prometheus/Loki).
In practice a lot of teams land on a hybrid: CloudWatch for AWS-native services because it’s automatic and cheap, DataDog for the application layer, with CloudWatch metrics ingested into DataDog for one unified view. It’s unglamorous, and it works.
Starting from scratch#
The order I’d do things: talk to developers about what hurts during incidents. Define SLOs for the flows that matter — what does “working” mean for login, checkout, search? Instrument that critical path first. Add distributed tracing next; it has the best debugging ROI in a microservice setup. Write a runbook for every alert so 3am-you knows what to check. Then put a quarterly review in the calendar to delete stale alerts, fix drifted thresholds, and confirm the dashboards still match the architecture.
And the traps I keep pulling teams out of: too many tools (three that work together beat six that don’t), graphs with no baseline (is 500 rps normal or a 10x spike?), monitoring things users don’t feel (disk I/O on a stateless container), and no redundancy for the monitoring itself — if your alerting dies during an incident, you’re blind exactly when you can least afford it.
None of this needs the fanciest tooling. It needs developers to get the information they need, in the place they’ll actually look, fast.