Separate durable HPC storage from the layer that accelerates a job

Source facts

Microsoft distinguishes an HPC dataset’s permanent storage from an optional acceleration layer that improves computational access. Its selection guide considers stored volume, CPU count, and file sizes together. For datasets below 50 TiB, the guide proposes Azure Files or Azure NetApp Files without an accelerator; it also gives that recommendation for 50–5,000 TiB with fewer than 500 cores. These are workload-selection guidelines, not a measured result for a particular application. Microsoft Learn.

Applicability

Use this distinction when reviewing an Azure VM HPC storage design. Describe the actual job’s dataset and access requirements before adopting a diagram intended for a different scale. Do not substitute a service name for a statement of which layer owns the durable copy.

DSE recommendation

Document the durable dataset location separately from any proposed performance accelerator, then qualify the combination with the workload’s file profile. Ask the designer to justify an accelerator against a simpler core-storage-only proposal. Include input staging, intermediate output, and final result placement in that comparison. Treat the source’s branches as starting points for evaluation, not mandatory architecture for every job.

Verification

Rehearse a representative job using the proposed arrangement. Record its file-size distribution, core count, data volume, and observed access behavior. Confirm where final results are retained before any temporary resources are removed. Keep a failed throughput target separate from a failed persistence requirement so the next design change addresses the actual gap.

Official references

Microsoft Learn: HPC storage best practices. Source reviewed September 9, 2026.