I'm newbie with this forum and this is first time to use Intel Micro Benchmark for testing FDR infiniband Perofrmance.
Anyway, i use intel micro benchmark 3.2.4 and I have problem about Allgather with 1024 processes and 4M message sizes.
# Benchmarking Allgather
# #processes = 1024
# ( 256 additional processes waiting in MPI_Barrier)
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.01 0.01 0.01
1 1000 104.29 104.40 104.33
2 1000 114.27 114.37 114.31
4 1000 136.66 136.75 136.71
8 1000 177.69 177.80 177.73
16 1000 238.50 238.66 238.60
32 1000 333.43 333.62 333.54
64 1000 697.89 698.26 698.05
128 1000 854.70 855.05 854.87
256 1000 930.07 930.48 930.27
512 1000 1090.33 1090.75 1090.52
1024 875 1642.99 1643.97 1643.47
2048 812 2324.69 2325.83 2325.25
4096 812 4665.67 4668.12 4666.91
8192 812 6475.55 6477.72 6476.66
16384 812 10368.64 10369.84 10369.25
32768 533 18721.53 18725.46 18723.87
65536 294 33836.31 33844.84 33840.93
131072 154 65191.53 65219.06 65204.86
262144 78 128504.25 128580.44 128542.72
524288 38 270415.55 270791.08 270577.44
1048576 14 598741.36 601155.43 600013.42
2097152 7 1259113.14 1267490.28 1263382.81
2 total processes killed (some possibly by mpirun during cleanup)
and Gatherv and Allgatherv(the result is quite similar) show about interger overflow
#----------------------------------------------------------------
# Benchmarking Gatherv
# #processes = 512
# ( 768 additional processes waiting in MPI_Barrier)
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.07 0.19 0.11
1 1000 62.69 91.45 78.84
2 1000 60.23 76.77 73.33
4 1000 70.90 87.93 85.35
8 1000 54.29 76.14 70.08
16 1000 62.76 81.96 76.67
32 1000 64.86 84.97 80.23
64 1000 73.83 93.81 88.77
128 1000 75.35 100.01 93.76
256 1000 83.21 112.15 103.90
512 1000 101.91 134.05 119.33
1024 1000 134.34 167.31 148.18
2048 1000 221.45 263.69 239.97
4096 1000 377.56 463.34 406.43
8192 1000 3299.10 3680.48 3571.74
16384 1000 2861.94 3275.25 3193.59
32768 808 4740.91 5192.17 5009.50
65536 623 3426.82 9136.07 7408.62
131072 320 11377.57 14969.00 12842.39
262144 55 90220.56 93014.00 92067.95
524288 55 77019.98 79891.46 78945.73
1048576 40 129610.60 136606.38 134520.82
2097152 20 239053.26 266906.06 259315.17
4194304 int-overflow.; The production rank*size caused int overflow for given sample
After I face the problem, i check dmesg command which it doesn't show anything about this problem and i saw the comment below Intel Micro Benchmark 3.2.4. He suggest about size_t in IMB_mem_manager.c --> r_len = c_info->num_procs*(size_t)init_size;. After i change this file and compile Intel Micro Benchmark, it still have same problem. I try older version (3.2.2 , 3.2.3 and 4.0.0 beta ) and still be same.
P.S. sorry for bad writing skill and I hope i will improve my skill more than this.
Thank you very much.