I am trying to use MPI in multiple MICs, and I get the DAPL error, The following is the info that I enable the MPI_DEBUG=5
wwu12:lips ~/work/mic/mpitest> mpirun -genv I_MPI_DAPL_PROVIDER_LIST=ofa-v2-scif0 -env I_MPI_DEBUG=5 -env I_MPI_MIC=enable -hostfile mpi_host -perhost 1 -n 2 /tmp/test.mic
Max MV2_DEFAULT_MAX_SG_LIST is 0, set to 1
Max MV2_SRQ_SIZE is 0, set to 512
Max MV2_DEFAULT_MAX_SG_LIST is 0, set to 1
Max MV2_SRQ_SIZE is 0, set to 512
[0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-scif0
[1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-scif0
[1] MPI startup(): DAPL provider ofa-v2-scif0
[0] MPI startup(): DAPL provider ofa-v2-scif0
[1] MPI startup(): dapl data transfer mode
[0] MPI startup(): dapl data transfer mode
[0:mic0] unexpected DAPL event 0x4003
Assertion failed in file ../../dapl_init_rc.c at line 1337: 0
There is no error when I run the program in a single MIC or in host and 1 MIC. Anyone know where the problem is?