Allow for duplicate notifications during Alertmanager HA partitions

Bottom line: DSE recommendation: include duplicate notifications in an Alertmanager HA failure exercise and agree how the receiving team will handle them. Keep that review separate from how alerts are grouped or muted, and avoid using an exactly-once assumption as the acceptance rule for a partition test.

Source fact: what the authority says

The Alertmanager HA guide describes a fail-open design that favors duplicate notifications over missed alerts during network partitions. It directs Prometheus to send alerts to every Alertmanager instance rather than through a load balancer. The guide distinguishes that behavior from strict exactly-once semantics.

What the source does not establish

These design statements do not prove that a particular alert reached an attended receiver or that a deployment cannot lose notifications. This article does not guarantee a duplicate count or delivery time. It does not approve a topology, cluster transport setting or receiver integration for an unreviewed installation.

Applicability questions

• Which Prometheus servers, Alertmanager members and receiver integrations are actually in the reviewed route?
• Which version and topology evidence will establish how that route was configured?
• Who will distinguish repeated delivery from separate incidents when reviewing the exercise?
• What authority, observation points and stop conditions are required for a safe failure rehearsal?

DSE recommendation: controlled next steps

The following paragraphs are DSE recommendations, not instructions or guarantees from the source.

Draw the reviewed alert path and identify its owners before planning a test. Record the intended cluster membership and the receiving destinations. Ask the team to explain how it will recognize the same test incident across multiple deliveries without hiding genuinely different affected services. Keep the incident identity and the membership of a notification group visible in the review.

Review the documented all-member delivery requirement against the actual configuration as its own task. Do not treat a functioning interface or a single successful notification as the complete topology review. Separate this work from decisions about grouping labels, silences and inhibition so a change in one area is not mistaken for evidence about another.

Agree the receiver workflow for duplicates before introducing a failure. Assign responsibility for observing notifications, recording timing and deciding when the exercise must stop. Keep any receiver automation in scope for its own review rather than assuming that repeated input is harmless. Preserve unexpected observations and unresolved delivery gaps for investigation instead of changing the acceptance rule to fit them.

Verification and evidence

Proposed verification, not a performed test: in an isolated or explicitly approved environment, define a synthetic alert and a bounded partition scenario. Record the starting membership and route, then observe each approved receiving destination. Compare the result with the documented design preference while recording actual delivery, duplication and missing evidence separately.

Retain configuration identifiers, versions, the approved failure plan, synthetic alert identity, observations from each member and receiver, and the team’s interpretation. Include the return-to-normal review in the same record. Test real collection and rule correctness separately. Do not describe a successful rehearsal as proof of every future network failure or an exactly-once delivery guarantee.

Official references

• Prometheus — High Availability — official product documentation; source version and applicability limits are stated above.

Source review completed October 28, 2026. Recheck the official documentation and deployment-specific applicability before a production change.