Review file granularity before scaling a Data Lake analytics pipeline

Source facts

Microsoft’s Data Lake guidance identifies per-file work such as listing, access checks and metadata operations as overhead for analytics engines. Many small files can therefore hurt processing performance.

Where raw inputs cannot be controlled, the guidance recommends aggregating them into larger downstream files and considering a read-oriented format such as Parquet. It also warns that some engines process files above 100 GB inefficiently, so larger is not an unlimited prescription. Microsoft Learn.

Applicability

Identify the raw arrival pattern, downstream engine, query shape and current file-size distribution. Keep raw capture requirements separate from the layout used for repeated analysis.

DSE recommendation

DSE recommends a representative aggregation experiment with explicit freshness and correctness requirements. Choose the target size and format with the downstream owner rather than copying a universal setting. Preserve a reconciliation path from the aggregated output back to the original inputs. Include late arrivals and failed aggregation runs in the design before replacing a production consumer’s input path.

Verification

Compare the same approved dataset and queries before and after the layout change. Record file count, processing duration, resource use and any missing or repeated records. Check that the consumer’s required freshness remains acceptable. Approve the change only when measured improvement and data reconciliation support it; a lower file count alone is not evidence that the resulting dataset is correct.

Official references

Microsoft Learn: Best Practices for Using Azure Data Lake Storage. Source retrieved September 9, 2026.