Most monitoring setups I’ve come across were built by an ops team, for an ops team. The developers — the people who actually write the bugs and have to fix them — were an afterthought. That’s backwards. Monitoring is a product, and developers are its users. If they’re not using it, it doesn’t matter how complete the coverage looks.
You can see the cost of getting this wrong without looking hard. Developers burning hours trying to reproduce issues they can’t see. Production failures announced by angry users rather than alerts. A wall of alerts with no context, so everything gets ignored. Incidents where the first chunk of time goes to working out what happened after the damage is done. It’s not just slower delivery — it grinds down morale, because nobody enjoys debugging blind.
What “good” actually means#
The three pillars — metrics, logs, traces — are table stakes. Metrics tell you something is wrong (CPU, memory, request rates, latency percentiles), logs tell you what happened, traces tell you where it started. The part that separates a good setup from a pile of tools is that the three are connected, so you can move from an alert to the logs to the trace without changing mindset or copy-pasting timestamps.
Dashboards should be built for the people reading them, not the people who made them. That means business metrics next to technical ones, obvious visual indicators of health, drill-down from overview to detail, and context like recent deployments and config changes right there on the page. A developer shouldn’t need to be a monitoring expert to answer “is my service okay?”
Alerting is where most setups fall over. Thresholds should come from baselines and anomaly detection, not numbers someone guessed in 2022. Issues should route to the team that owns them. Related alerts should be grouped, not fired individually fifty times. And every alert should answer “what’s broken?” and at least gesture at “where do I start?”. Finally, the loop has to be fast — the gap between a problem occurring and a developer knowing should be seconds, which takes real-time metrics, streaming searchable logs, tracing across service boundaries, and deploy markers so releases can be correlated with trouble.
Do the unglamorous groundwork#
Before picking tools, shadow developers during a debugging session. Note the questions they ask over and over, where the current process hurts, and which metrics actually track product success. Then lower the barrier to entry: libraries and SDKs that make instrumentation trivial, templates for the common cases, self-service dashboards, and docs that explain why, not just how.
And treat monitoring as a system that has to evolve: all configuration in infrastructure-as-code, alert definitions and dashboards in version control, automated tests for monitoring rules, and a plan for multi-region or multi-cloud before you’re forced into one.
The stack, concretely#
Names change, but the shape of a modern stack is fairly stable:
- Metrics: a time-series database — Prometheus, InfluxDB, or TimescaleDB — plus an APM tool, plus custom metrics out of your business logic.
- Logs: centralised aggregation (ELK Stack, Splunk, or Loki), structured logging standards so the logs are queryable, and retention policies that balance cost against usefulness.
- Traces: OpenTelemetry for instrumentation, something like Jaeger, Zipkin, or Honeycomb to visualise, and a sampling strategy once volume gets serious.
- Synthetics: uptime checks from multiple regions, scripted user journeys, API health checks.
- Real User Monitoring: frontend performance, user-experience metrics, and error tracking from actual production traffic — because your synthetic canary doesn’t use a three-year-old Android phone on hotel wifi.
Small team on AWS: CloudWatch first#
For a small team on AWS, CloudWatch is the fastest route to something genuinely useful. EC2, Lambda, RDS and ECS report to it with zero configuration, so you get infrastructure visibility without writing instrumentation. You pay for what you use — typically $10–50 a month at small scale, no fixed platform fee — and metrics, logs, traces (via X-Ray) and alarms live in one place. You can have meaningful alerts and dashboards in hours.
Getting the most out of it:
- CloudWatch Logs Insights replaces a lot of what people stand up Elasticsearch for. A query like
fields @timestamp, @message | filter @message like /ERROR/ | stats count() by bin(5m)answers “are errors trending up?” in one line. - Composite alarms keep the noise down — alert when error rate is high and response time is degraded, not on each condition alone.
- Dashboards that mix business and technical metrics earn their keep. Put one on a TV, or drop the link in Slack.
- Custom metrics via the agent or SDK are where the real value is: sign-ups, payments, feature usage, next to the infrastructure numbers.
- CloudWatch Synthetics runs scheduled canaries through your critical paths, so you hear about a broken checkout before users do.
- X-Ray gives you distributed tracing with minimal setup, and the CloudWatch integration ties metrics to request-level traces.
You’ll outgrow it eventually: past roughly 50 engineers, going multi-cloud, needing proper ML-based anomaly detection, or when dashboard customisation and alerting logic hit the ceiling of what the UI allows.
At enterprise scale: DataDog#
Once you’re running hundreds of services across multiple environments, DataDog is the de facto choice, and mostly deserves to be. It watches everything in one view — AWS, Azure, GCP, on-prem, containers, serverless, databases, third-party services, frontend. Watchdog uses ML to flag anomalies and surface probable root causes, which matters because no human is watching thousands of services. The collaboration layer is the other half of the value: team-specific dashboards, role-based access control, shared investigation notebooks, and integrations with PagerDuty and Opsgenie. Alerting goes as deep as you need — multi-condition boolean logic, forecasting alerts that fire before a threshold is breached, anomaly detection that adapts to traffic patterns, maintenance windows. And the APM reaches code-level profiling, security monitoring tied to traces, cost attribution per service, and auto-generated service maps.
Rolling it out across a large organisation takes actual planning. Phase the adoption — critical services first, organised with a proper tagging strategy by team, environment and business unit. Set standards early: metric and tag naming, dashboard templates per service type, alert severity levels and escalation paths, SLO definitions per tier. Wire it into everything — CI/CD for deploy markers, incident management, Slack or Teams, ITSM ticketing. Invest in enablement: internal docs, a champion on each team, workshops on APM and profiling. And manage the bill deliberately, because at enterprise level it’s $20–100K+ a year: filter out noisy low-value metrics, sample traces on high-volume services, audit feature usage, and tag for cost allocation.
The payoff, when it works, is faster incident resolution thanks to the correlation features, problems caught before users notice, developers self-serving instead of queueing behind an ops team, and enough visibility into resource usage to right-size infrastructure — which sometimes covers the licence by itself.
Alternatives worth pricing up: New Relic (similar capability, sometimes cheaper for high-volume tracing), Dynatrace (strong AIOps, common in large financial firms), Splunk (unmatched log analysis if security already runs it), and Grafana Cloud (the natural step if you’re on Prometheus/Loki).
Plenty of organisations run a hybrid and are happy: AWS-native services report to CloudWatch automatically, DataDog covers the applications and everything outside AWS, and DataDog ingests the CloudWatch metrics so there’s still one view. It balances cost and capability without drama.
Culture is the other half#
The tooling fails without habits around it. Treat instrumentation as production code: it shows up in code review, custom metrics get tests, and monitoring decisions get written down in architecture discussions. Treat incidents as monitoring feedback: run blameless postmortems and always ask “what monitoring would have caught this sooner?”, then actually update the dashboards and alerts. And audit quarterly — delete unused dashboards and alerts, adjust thresholds as the system evolves, verify alerts still fire, keep the docs honest.
The pitfalls are predictable, and I’ve seen every one of them: adopting every shiny tool instead of a few that integrate; graphs with no baseline or trend, so nobody knows if a number is good; a technically excellent system that fails because nobody was trained on it; metrics collected out of curiosity rather than because they matter to users; and no redundancy for the monitoring itself, so you go blind at the exact moment you need eyes.
If you’re starting a monitoring overhaul, the sequence I’d follow: audit the current gaps and pain points, define SLOs and SLIs around user experience, instrument the critical user journeys first, add distributed tracing (the biggest debugging win in a microservice world), link every alert to a runbook, and measure whether it’s working — MTTR, deployment frequency, and whether developers actually say their lives got easier.
That last one is the real test. Good DevOps work doesn’t just keep systems up; it makes the development team faster and calmer. Monitoring that developers open voluntarily is one of the clearest signs you’ve done it.