When one upstream fault trips dozens of monitors, group the evidence around what viewers lost and give master control one decision to make.
A shared decoder stops delivering usable video. Downstream probes report black, silence and missing captions across several outputs. A separate feed in the same region loses its audio at nearly the same time. The monitoring desk needs to group the cascade around the decoder and keep the unrelated audio incident visible. Grouping by time or geography alone can turn two faults into one misleading page.
Why it matters. Every notification asks an operator to stop and decide. During a cascade, the priority is to identify the affected services, find a working recovery path and assign ownership. Google’s 2016 SRE guidance recommends paging on symptoms or imminent problems and collecting diagnostic detail separately. In master control, an impaired program output is the symptom. An input or decoder fault can explain it and help pick the next action. Google SRE, Monitoring Distributed Systems.
What should wake the operator?
Page for a decision that cannot wait. Define the viewer symptom, affected output and action window before choosing a severity. An impaired primary with a verified, ready backup calls for a source decision. The loss of the only remaining backup may also warrant a page during a protected event, even while the primary stays clean. A corrected packet error on a redundant path with healthy output can go to an engineering queue under a different policy. ETSI’s June 2020 DVB measurement guidance provides transport-stream checks. Use those results as evidence and set page urgency against the service’s exposure and recovery options. ETSI TR 101 290 V1.4.1.
How do you group a cascade without hiding a second fault?
Require a known dependency and corroborating observations. Record which services used each input, decoder or processing node when the alarms fired. For the decoder example, show the shared decoder fault and its affected outputs together. Keep each raw alarm and its timestamp under the incident. Leave the unrelated audio loss separate because its signal path bypassed the decoder. If a route is unknown or the timing is ambiguous, flag the possible relationship for the operator and keep both incidents visible.
Prometheus Alertmanager documents grouping related alerts and inhibiting notifications while a matching parent alert is active. Apply inhibition to notifications within the confirmed scope. Reevaluate children when the parent clears or a route changes. A still-silent output needs its own recovery decision. Prometheus Alertmanager, checked November 5, 2026.
How can a shift tell whether triage improved?
Replay a noisy shift and score both speed and misses. For each replay, log pages per confirmed incident, time to the first correct owner, time to first useful action and affected outputs absent from the incident card. Include a concurrent independent fault, a probe outage and a partial recovery. Preserve unknown states when the probe itself fails. Compare against the existing rules at equal incident coverage. Google’s 2018 SRE Workbook evaluates alerting against precision, recall, detection time and reset time. A quieter pager is valuable only if operators still see the faults they need to act on. Google SRE Workbook, Alerting on SLOs.
Start with one service family whose route history and ownership are reliable. Give its lead operator a compact card with the affected outputs, last known good time, suspected dependency and a link to decoded evidence. Review which pages would have changed before applying the rule to another service family. For teams also validating downstream quality, the live caption QC guide shows why a healthy transport check is only one piece of service evidence.
