I am testing Intel MPI 4.1 with test.c (the provided test program).
Whenever I run > 2000 ranks the program executes correctly but fails to end gracefully.
Running:
mpiexec.hydra -n 2001 -genv I_MPI_FABRICS shm:ofa -f hostfile ./testc
It stalls at
...
....
Hello World: Rank 2000 running on host xxxx
##<stalls here; does not return to command prompt>
(If I use -n 2000 or less, it runs perfectly.)
I have testing 3000 ranks using OpenMPI, so it doesn't seem to be a cluster/network issue.