{ "api_version": "1", "kind": "dse_post", "self": "https://update.dsesecurity.com/api/v1/posts/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "item": { "id": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "slug": "dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation", "url": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "alternate_urls": { "markdown": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation.md", "json": "https://update.dsesecurity.com/api/v1/posts/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/" }, "title": "Expect batch-local totals from Azure Monitor pipeline aggregation", "summary": "Why can a pipeline aggregation emit more than one result for the same time bin?", "format": { "slug": "explainer", "name": "Explainer" }, "priority": { "slug": "info", "name": "Information" }, "featured": false, "image": { "theme": "managed-it", "label": "Managed IT operations", "alt": "A controlled technology lifecycle progressing from assessment to approved production.", "card_url": "https://update.dsesecurity.com/assets/editorial/managed-it-card.webp?v=1.8.20", "hero_url": "https://update.dsesecurity.com/assets/editorial/managed-it-hero.webp?v=1.8.20", "social_url": "https://update.dsesecurity.com/assets/editorial/managed-it-social-v2.jpg?v=1.8.20", "width": 2400, "height": 1350 }, "topics": [ { "slug": "it", "name": "IT", "url": "https://update.dsesecurity.com/topic/it/" } ], "author": { "name": "DSE Security Editorial Team", "url": "https://update.dsesecurity.com/#editorial-team", "type": "Organization" }, "publisher": { "name": "Detection Systems & Engineering", "url": "https://dsesecurity.com/" }, "published_at": "2026-09-10T00:22:04+00:00", "modified_at": "2026-09-10T02:14:32+00:00", "reviewed_on": "2026-09-09", "reading_minutes": 2, "word_count": 245, "potentially_affected": "Azure Monitor pipeline dataflows that aggregate logs with summarize before cloud ingestion.", "dse_recommendation": "Design downstream interpretation around batch-local aggregates rather than assuming one final record per time bin.", "primary_source": { "name": "Configure Azure Monitor pipeline transformations - Azure Monitor | Microsoft Learn", "url": "https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/pipeline-transformations", "published_on": null, "authority": "Microsoft Learn" }, "publishing_principles": "https://update.dsesecurity.com/updates/dse-updates-editorial-methodology/", "usage_info": "https://update.dsesecurity.com/usage/", "copyright_notice": "Copyright © 2026 Detection Systems & Engineering. All rights reserved.", "content_html": "

Source facts

\n

Azure Monitor pipeline aggregates each incoming batch; its default processing interval is one minute. A batch processor is required, and changing that interval is not available through the portal. Microsoft warns that aggregation adds up to five minutes of latency and that a query using bin() can produce multiple records for the same interval because ingestion is streaming and batched. Microsoft Learn.

\n

Applicability

\n

Review dataflows using summarize, their batch-processor configuration, and the queries consuming the resulting records. Keep the aggregation window separate from the time bins expressed in KQL. Establish what each exported value represents before treating repeated timestamps as duplicate source events.

\n

DSE recommendation

\n

Design downstream interpretation around batch-local aggregates rather than assuming one final record per time bin. Have the query owner document how multiple partial results will be handled for the chosen calculation. Retain the original count or other necessary supporting fields in the proposed schema when the downstream calculation needs them. Review latency tolerance before using these aggregates for urgent decisions.

\n

Verification

\n

Send a controlled set of records across more than one collection batch while keeping their event times inside one selected bin. Compare emitted aggregates with the known inputs and the downstream result. Check timing as well as arithmetic. Preserve the batch configuration, transformation, input set, and outputs so a later interval change can be assessed without assuming that matching bin labels imply a single completed total.

\n

Official references

\n

Microsoft Learn: Azure Monitor pipeline transformations. Source reviewed September 9, 2026.

", "content_text": "Source facts\nAzure Monitor pipeline aggregates each incoming batch; its default processing interval is one minute. A batch processor is required, and changing that interval is not available through the portal. Microsoft warns that aggregation adds up to five minutes of latency and that a query using bin() can produce multiple records for the same interval because ingestion is streaming and batched. Microsoft Learn.\nApplicability\nReview dataflows using summarize, their batch-processor configuration, and the queries consuming the resulting records. Keep the aggregation window separate from the time bins expressed in KQL. Establish what each exported value represents before treating repeated timestamps as duplicate source events.\nDSE recommendation\nDesign downstream interpretation around batch-local aggregates rather than assuming one final record per time bin. Have the query owner document how multiple partial results will be handled for the chosen calculation. Retain the original count or other necessary supporting fields in the proposed schema when the downstream calculation needs them. Review latency tolerance before using these aggregates for urgent decisions.\nVerification\nSend a controlled set of records across more than one collection batch while keeping their event times inside one selected bin. Compare emitted aggregates with the known inputs and the downstream result. Check timing as well as arithmetic. Preserve the batch configuration, transformation, input set, and outputs so a later interval change can be assessed without assuming that matching bin labels imply a single completed total.\nOfficial references\nMicrosoft Learn: Azure Monitor pipeline transformations. Source reviewed September 9, 2026.", "content_markdown": "## Source facts\n\nAzure Monitor pipeline aggregates each incoming batch; its default processing interval is one minute. A batch processor is required, and changing that interval is not available through the portal. Microsoft warns that aggregation adds up to five minutes of latency and that a query using bin() can produce multiple records for the same interval because ingestion is streaming and batched. [Microsoft Learn](https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/pipeline-transformations).\n\n## Applicability\n\nReview dataflows using summarize, their batch-processor configuration, and the queries consuming the resulting records. Keep the aggregation window separate from the time bins expressed in KQL. Establish what each exported value represents before treating repeated timestamps as duplicate source events.\n\n## DSE recommendation\n\nDesign downstream interpretation around batch-local aggregates rather than assuming one final record per time bin. Have the query owner document how multiple partial results will be handled for the chosen calculation. Retain the original count or other necessary supporting fields in the proposed schema when the downstream calculation needs them. Review latency tolerance before using these aggregates for urgent decisions.\n\n## Verification\n\nSend a controlled set of records across more than one collection batch while keeping their event times inside one selected bin. Compare emitted aggregates with the known inputs and the downstream result. Check timing as well as arithmetic. Preserve the batch configuration, transformation, input set, and outputs so a later interval change can be assessed without assuming that matching bin labels imply a single completed total.\n\n## Official references\n\n[Microsoft Learn: Azure Monitor pipeline transformations](https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/pipeline-transformations). Source reviewed September 9, 2026." }, "json_ld": { "@context": "https://schema.org", "@graph": [ { "@type": "Organization", "@id": "https://dsesecurity.com/#organization", "name": "Detection Systems & Engineering", "alternateName": "DSE Security", "url": "https://dsesecurity.com/", "logo": { "@type": "ImageObject", "url": "https://update.dsesecurity.com/assets/dse-logo-20260812.png?v=1.8.20" } }, { "@type": "Organization", "@id": "https://update.dsesecurity.com/#editorial-team", "name": "DSE Security Editorial Team", "url": "https://update.dsesecurity.com/", "parentOrganization": { "@id": "https://dsesecurity.com/#organization" } }, { "@type": "WebSite", "@id": "https://update.dsesecurity.com/#website", "name": "DSE Updates", "alternateName": "DSE Security Knowledge Hub", "url": "https://update.dsesecurity.com/", "inLanguage": "en-US", "publisher": { "@id": "https://dsesecurity.com/#organization" }, "potentialAction": { "@type": "SearchAction", "target": { "@type": "EntryPoint", "urlTemplate": "https://update.dsesecurity.com/?q={search_term_string}" }, "query-input": "required name=search_term_string" } }, { "@type": "WebPage", "@id": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "url": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "isPartOf": { "@id": "https://update.dsesecurity.com/#website" }, "lastReviewed": "2026-09-09" }, { "@type": "BreadcrumbList", "@id": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/#breadcrumbs", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "DSE Updates", "item": "https://update.dsesecurity.com/" }, { "@type": "ListItem", "position": 2, "name": "Expect batch-local totals from Azure Monitor pipeline aggregation", "item": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/" } ] }, { "@type": [ "Article", "TechArticle" ], "@id": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/#article", "identifier": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "url": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/", "headline": "Expect batch-local totals from Azure Monitor pipeline aggregation", "description": "Why can a pipeline aggregation emit more than one result for the same time bin?", "abstract": "Why can a pipeline aggregation emit more than one result for the same time bin?", "articleBody": "Source facts\nAzure Monitor pipeline aggregates each incoming batch; its default processing interval is one minute. A batch processor is required, and changing that interval is not available through the portal. Microsoft warns that aggregation adds up to five minutes of latency and that a query using bin() can produce multiple records for the same interval because ingestion is streaming and batched. Microsoft Learn.\nApplicability\nReview dataflows using summarize, their batch-processor configuration, and the queries consuming the resulting records. Keep the aggregation window separate from the time bins expressed in KQL. Establish what each exported value represents before treating repeated timestamps as duplicate source events.\nDSE recommendation\nDesign downstream interpretation around batch-local aggregates rather than assuming one final record per time bin. Have the query owner document how multiple partial results will be handled for the chosen calculation. Retain the original count or other necessary supporting fields in the proposed schema when the downstream calculation needs them. Review latency tolerance before using these aggregates for urgent decisions.\nVerification\nSend a controlled set of records across more than one collection batch while keeping their event times inside one selected bin. Compare emitted aggregates with the known inputs and the downstream result. Check timing as well as arithmetic. Preserve the batch configuration, transformation, input set, and outputs so a later interval change can be assessed without assuming that matching bin labels imply a single completed total.\nOfficial references\nMicrosoft Learn: Azure Monitor pipeline transformations. Source reviewed September 9, 2026.", "datePublished": "2026-09-10T00:22:04+00:00", "dateModified": "2026-09-10T02:14:32+00:00", "mainEntityOfPage": { "@id": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/" }, "inLanguage": "en-US", "isAccessibleForFree": true, "author": { "@type": "Organization", "name": "DSE Security Editorial Team", "url": "https://update.dsesecurity.com/#editorial-team" }, "publisher": { "@id": "https://dsesecurity.com/#organization" }, "image": { "@type": "ImageObject", "@id": "https://update.dsesecurity.com/updates/dse-20260909-592-expect-batch-local-totals-from-azure-monitor-pipeline-aggregation/#primaryimage", "url": "https://update.dsesecurity.com/assets/editorial/managed-it-social-v2.jpg?v=1.8.20", "contentUrl": "https://update.dsesecurity.com/assets/editorial/managed-it-social-v2.jpg?v=1.8.20", "width": 1200, "height": 630, "caption": "Expect batch-local totals from Azure Monitor pipeline aggregation" }, "articleSection": [ "IT" ], "keywords": [ "IT", "Explainer", "Information priority" ], "genre": "Explainer", "about": [ { "@type": "Thing", "name": "IT", "url": "https://update.dsesecurity.com/topic/it/" } ], "wordCount": 245, "timeRequired": "PT2M", "publishingPrinciples": "https://update.dsesecurity.com/updates/dse-updates-editorial-methodology/", "usageInfo": "https://update.dsesecurity.com/usage/", "copyrightHolder": { "@id": "https://dsesecurity.com/#organization" }, "copyrightNotice": "Copyright © 2026 Detection Systems & Engineering. All rights reserved.", "citation": { "@type": "CreativeWork", "name": "Configure Azure Monitor pipeline transformations - Azure Monitor | Microsoft Learn", "url": "https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/pipeline-transformations" } } ] } }