Size an MPI transport experiment by connection count, not node count alone

Source facts

Microsoft’s Azure HPC guidance estimates an MPI job’s maximum connections by multiplying processes per node by the square of the job’s node count. It suggests UCX_TLS=rc,sm for jobs below 256K connections and UCX_TLS=dc,sm above 256K connections. The source also emphasizes workload-specific tuning experiments rather than assuming an optimal configuration from VM selection alone. Microsoft Learn.

Applicability

Use this planning check when scaling an MPI job beyond an earlier test. Record both the process layout and node count; a node-count-only label does not express the source’s connection estimate. The documented smaller/larger examples do not specify the exact boundary case.

DSE recommendation

Record the estimated connection count alongside the proposed MPI transport experiment. Have the application owner compare the intended job layout with the transport guidance for its actual MPI runtime. Keep the runtime, transport setting, process placement and input workload together in the experiment record. Avoid carrying an old small-job setting into a larger run without examining that estimate.

Verification

Rehearse the candidate configuration with an approved representative job. Check correctness first, then compare completion behavior and measured scaling against the prior configuration. Change one intended variable at a time where practical so an improvement or regression can be attributed. Retain an unsuccessful result as useful evidence instead of turning the source’s tuning suggestion into a claimed performance guarantee.

Official references

Microsoft Learn: Scaling HPC applications. Source reviewed September 9, 2026.