The dashboard is green while the heartbeat has stopped
A worker normally publishes a custom heartbeat metric every minute. After a deployment it stops publishing, but the alarm was configured to treat missing points as not breaching. Queue age is climbing. The team has not yet identified whether the worker or metric publishing failed.
Your taskWhich alarm and investigation decisions would avoid treating missing telemetry as proof of health?
Review the explained decision
Reasoned approach. For a continuously emitted heartbeat, investigate missing data as a potentially bad signal and choose alarm missing-data behavior consistent with that expectation. Correlate the stopped heartbeat with queue age, worker logs and deployment events before changing capacity or rolling back. Test the alarm behavior by withholding sample metric data in an authorized environment.
Why the shortcut fails. Lowering the heartbeat threshold cannot make absent data appear. Treating every sparse error-only metric as breaching would create false alarms, so missing-data policy must match the metric's expected publication pattern.
Check your work. Write the expected alarm state for a missed heartbeat and for an error-only metric with no errors. Record a responder and an escalation path for the queue-age signal.
AWS: CloudWatch missing data