Preserve a Prometheus discovery label before filtering metrics with it

Source facts

Microsoft states that discovered targets carry monitor-specific meta-labels and that labels beginning with double underscores are dropped after discovery. To filter metrics using that information, its guide first preserves the value with relabelings, then filters with metricRelabelings. The latter stage runs after scraping and before ingestion. Microsoft Learn.

Applicability

Use this review for an Azure Managed Prometheus PodMonitor or ServiceMonitor filter that depends on target-discovery metadata. Identify the original meta-label, the ordinary label chosen to preserve its value and the intended included targets. Keep this question separate from API migration or alert-rule deployment.

DSE recommendation

DSE recommends reviewing the two transformation stages together before changing collection. Write a small expected-result table containing an included target and an excluded target. Check that the preservation rule produces the exact label read by the later filter. Have the application owner confirm the excluded metrics are intentional before applying the change.

Verification

In a controlled target set, inspect discovery results and the resulting ingested labels. Confirm the preserved value exists where the metric filter expects it, then compare included and excluded series with the table. Investigate a missing target separately from a filtered metric. Retain sanitized label examples and the deployed monitor configuration with the outcome.

Official references

Microsoft Learn: Create custom Prometheus scrape job from your Kubernetes cluster using CRDs.