Building a Reusable Observability and Alerting Layer for AWS Lambda

When something breaks, whoever is on call needs three things in quick succession: whether it failed, how long it has been broken, and what to actually do about it. Most Lambda setups answer the first question well and leave the other two to guesswork. This client already had basic CloudWatch metrics and a simple Grafana dashboard covering alerts, so the first version of this project focused on turning that foundation into a proper multi-channel alerting layer rather than standing up a new metrics backend. Sentry.io came later, once the client asked for deeper error triage. It fit into the design without requiring a rebuild, which is a good sign that the original design was sound.

That structure sounds simple, and it is meant to. The value is not the pipeline itself. It is that every function built on top of it fails the same way, gets classified the same way, and logs the same way, so an engineer moving between functions does not have to relearn how failure works each time.

Failures we consider retryable get re-raised, which lets the underlying platform or queue redeliver the event on its own schedule. Failures we consider non-retryable, such as a malformed payload or an unauthorized caller, return a success-shaped response to the platform and route the event to a dead-letter path instead. That distinction matters. Retrying a malformed payload five times does nothing but waste invocations and delay the moment someone notices the real problem, so we would rather remove it from the retry queue immediately and put it somewhere a human can review it.

The breaker only counts transient, timeout, and unknown failures against it. Schema errors, authorization failures, and duplicates do not trip it, because those failures say something about the request, not about whether the downstream system is healthy. Tripping the breaker over a malformed request would be like flipping the whole house’s power off because one lamp has a bad bulb.

Idempotency works alongside the breaker using the same pattern: a pluggable backend, DynamoDB in production and an in-memory store for local development and tests. Every message is deduplicated by its message_id with a time-to-live attached, so a message that arrives twice, which happens more often than most teams expect in distributed systems, gets acknowledged quietly the second time instead of being logged and alerted on as a failure.

For functions where p99 latency actually matters, we added an opt-in CloudWatch Embedded Metric Format integration. EMF publishes a real metric value, with full dimensionality, directly from the code the moment the phase completes, rather than requiring a separate metric filter pattern to be written and maintained for every value worth tracking. That made it the more direct way to get a trustworthy p99 duration metric per function, especially once we wanted that number broken down by failure type as well. It is opt-in per function rather than default, since most functions in this system do not need percentile-level latency tracking and there is no reason to pay for what nobody is going to look at.

Each phase also runs under its own SIGALRM-based timeout, a per-phase timer that fires if that specific phase runs long, so a single hung call inside EXECUTION cannot quietly consume the entire invocation without anyone knowing which phase actually stalled. All of this, the phase timeouts, the circuit breaker thresholds, and the EMF opt-in, is driven by a single WrapperConfig, so changing behavior is a configuration change rather than a code change.

Grafana comes pre-wired with all three datasources and a ready-made dashboard covering invocation rate, success rate, failures broken down by type, phase duration at the 99th percentile, circuit breaker events, and a live log stream with links that jump straight from a log line to its matching trace.

Ten scripted test scenarios exercise every code path the wrapper supports: the happy path, a schema error, a missing environment variable, an unauthorized source, a transient failure, an escalation after retries are exhausted, a permanent error, a duplicate delivery, a circuit trip, and a circuit recovery. Each one runs independently, for example SCENARIO=circuit docker compose run --rm test-runner, so an engineer can reproduce any single failure mode on demand instead of waiting for it to happen in production. A companion file, OBSERVABILITY-QUERIES.md, holds copy-paste Loki, PromQL, and Tempo queries for walking through the stack live, which turned out to be useful for demonstrating the system to the client directly, not just for our own debugging.

The fix follows a pattern AWS itself recommends: only the per-function composite alarm carries the actual notification actions. The individual alarms underneath it, covering failure rate, dead letters, init failures, circuit breaker trips, and latency, feed the composite alarm’s state logic but stay silent on their own. The same incident now caps at eight messages no matter how many of the underlying conditions fire together, and that ceiling holds even as we add more individual alarms underneath the same composite alarm later.

Alerts resolve themselves. CloudWatch flips an alarm from ALARM back to OK automatically once the underlying metric recovers, and the recovery notification pulls from the alarm’s own history to report how long the condition was actually active, with no manual close step required from whoever responded.

The same runbook text is shared across all four notifiers and embedded directly in the alert itself, so whoever gets paged sees not just what failed, but whether it is being retried automatically, whether it landed in a dead-letter queue, and what to actually do next, without leaving Slack, Teams, or their inbox. Fast-page alarms require two consecutive breaching periods before firing, roughly a two-minute detection delay, which is tunable per deployment and exists specifically to keep a metric bouncing near a threshold from turning into a notification storm. The opt-in p99 latency alarms mentioned earlier fold into this same composite pattern rather than getting a separate notification path of their own.

The whole thing is defined in Terraform, driven by a for_each over a list of functions, so wiring alerting up for a new Lambda already in the pattern takes zero new code, just one new entry in a variables map. All four notifier Lambdas have a local, no-deploy test harness, and we documented the exact aws cloudwatch set-alarm-state commands needed to force an alarm to fire or resolve on demand, so testing the alerting path never has to wait for a real failure to happen.

The wrapper’s ON_FAILURE path now also reports to Sentry alongside its existing structured logging, carrying full stack traces, a breadcrumb for each phase the invocation passed through, and the same failure_type, correlation_id, and lambda_name tags used everywhere else in the system. That closes a gap plain CloudWatch logs cannot: grouping and deduplicating recurring errors by fingerprint, watching error frequency and regression trends over time, and jumping straight to the offending line and variable state instead of parsing a JSON log line by hand. Because a Sentry issue carries the same correlation ID as the matching Loki log line and Tempo trace, an engineer can move from a Sentry issue straight to the exact trace and log line that produced it.

The net effect is a clean split of concerns. The four chat and email notifiers stay the fast, low-friction first alert. CloudWatch alarm history remains the lightweight record of when something went ALARM and back to OK. Sentry is where an engineer actually goes to debug once they are looking at the failure. All of that reads from the same composite alarm path and the same failure taxonomy, so nothing earlier in the stack had to change to support the addition.

The explicit failure taxonomy turned out to be the real backbone of the whole system. The same seven categories that drive the wrapper’s retry behavior also drive the alerting runbook text, which means there is one source of truth feeding two systems instead of two definitions that can quietly drift apart. The pluggable backend interfaces behind the circuit breaker and the idempotency store meant the same wrapper code runs against DynamoDB in production and an in-memory store in tests, with no conditional logic scattered around to tell the two apart. And because the local environment lets an engineer exercise all ten failure modes and watch the traces, logs, and metrics correlate in Grafana before ever touching AWS, most confidence about how this system behaves gets built before deployment rather than after an incident.

The pattern across this whole engagement was addition, not rework. We started with a wrapper that gave every function consistent logging, tracing, retries, and failure handling. We added alerting on top of it using only what CloudWatch already provides, because that was what the client actually needed at the time. When the client’s needs grew past chat notifications, Sentry attached to the same failure taxonomy and correlation model without touching anything that came before. If your Lambda functions each handle failure a little differently and your on-call process depends on someone remembering which function does what, then that is usually a sign the foundation is missing, not that your team needs to try harder. Are your Lambda functions telling you when they fail, how long they were broken, and what to do about it, or just the first of those three? If you want a second opinion on that, then will you share what you are running through our contact page?