Hi all,
I am new to HPC, playing with MKL, CUDA and HPL from Nvidia to optimize the result.
Everything compiling smoothly until enter the ptest/ directory and broke, It says:
make[2]: Entering directory `/root/hpl-2.0_FERMI_v15/testing/ptest/CUDA' mpicc -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/root/hpl-2.0_FERMI_v15/include -I/root/hpl-2.0_FERMI_v15/include/CUDA -I/opt/intel/mkl/include -I/opt/intel/impi/4.1.3.049/intel64/include -I/usr/local/cuda/include -fomit-frame-pointer -O3 -funroll-loops -W -Wall -fopenmp -mt_mpi -m64 -lmpi_mt -o /root/hpl-2.0_FERMI_v15/bin/CUDA/xhpl HPL_pddriver.o HPL_pdinfo.o HPL_pdtest.o /root/hpl-2.0_FERMI_v15/lib/CUDA/libhpl.a -L /root/hpl-2.0_FERMI_v15/src/cuda -ldgemm -L/opt/cuda6/lib64 -lcuda -lcudart -lcublas -L/opt/intel/mkl/lib/intel64 -lpthread -lmpi_mt /opt/intel/impi/4.1.3.049/intel64/lib/libmpi_mt.so -lmpi_mt /usr/lib64/gcc/x86_64-slackware-linux/4.8.2/../../../../x86_64-slackware-linux/bin/ld: MPIR_Thread: TLS definition in /opt/intel/impi/4.1.3.049/intel64/lib/libmpi_mt.so section .tbss mismatches non-TLS definition in /opt/intel/impi/4.1.3.049/intel64/lib/libmpi.so.4 section .bss /opt/intel/impi/4.1.3.049/intel64/lib/libmpi.so.4: could not read symbols: Bad value collect2: error: ld returned 1 exit status make[2]: *** [dexe.grd] Error 1
FYI:
mpicc -show
gcc -I/opt/intel/impi/4.1.3.049/intel64/include -L/opt/intel/impi/4.1.3.049/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/impi/4.1.3.049/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.1 -lmpigf -lmpi -lmpigi -ldl -lrt -lpthread
SHELL env
CC=/opt/intel/composer_xe_2013_sp1.3.174/bin/intel64/icc CXX=/opt/intel/composer_xe_2013_sp1.3.174/bin/intel64icpc F77=/opt/intel/impi/4.1.3.049/intel64/bin/mpiifort FC=/opt/intel/impi/4.1.3.049/intel64/bin/mpiifort FC90=/opt/intel/impi/4.1.3.049/intel64/bin/mpiifort LD_LIBRARY_PATH=/opt/intel/composerxe/mkl/lib/intel64:/opt/intel/impi/4.1.3.049/lib64:/opt/intel/composerxe/lib/intel64/:/opt/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/:/opt/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64/:/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/:/opt/cuda6/lib64:/opt/cuda6/lib/:/usr/local/bin/mpi/intel/lib/:/usr/X11R6/lib64/:/usr/local/lib64/ PATH= /opt/intel/impi/4.1.3.049/bin64/:/opt/intel/composer_xe_2013_sp1.3.174/bin/intel64:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib64/kde4/libexec:/usr/lib64/java/bin:/usr/lib64/java/jre/bin:/usr/lib64/java/jre/bin:/usr/lib64/qt/bin:/usr/share/texmf/bin
gcc --version
gcc (GCC) 4.8.2
part of Make.CUDA
MPdir = /opt/intel/impi/4.1.3.049/intel64 MPlib = $(MPdir)/lib/libmpi_mt.so LAdir = /opt/intel/mkl/lib/intel64 LAinc = -I/opt/intel/mkl/include LAlib = -L $(TOPdir)/src/cuda -ldgemm -L/opt/cuda6/lib64 -lcuda -lcudart -lcublas -L$(LAdir) -lpthread -lmpi_mt CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall -fopenmp -mt_mpi -m64 LINKFLAGS = $(CCFLAGS) -lmpi_mt
My `effort' on fix this error:
As what I found in:
https://software.intel.com/en-us/forums/topic/392483
I put -lmpi_mt everywhere to make sure no lib is NOT compiled with libmpi.so
In thread:
https://software.intel.com/en-us/forums/topic/294642
I put an extra -mt_mpi flag to CCFLAGS
I use the -show trick discribed here:
https://software.intel.com/en-us/forums/topic/508632
It display the -lmpi_mt, but the compiling still not working as well.
Tried to replace Impi with openmpi which compiled by icc as discribe on a web source pdf: HOWTO-HPL-GPU.pdf
the same problem as the top of this thread discribed
FYI2:
I used to succeed in compiling & running the (pure) hpl with MKL, part of Make.Linun_PII_CBLAS here:
HPLlibHybrid = /opt/intel/composer_xe_2013_sp1.3.174/mkl/benchmarks/mp_linpack/lib_hybrid/mic/libhpl_hybrid.a LAdir = /opt/intel LAinc = -I$(LAdir)/mkl/include LAlib = -L$(LAdir)/mkl/lib/intel64 -Wl,--start-group $(LAdir)/mkl/lib/intel64/libmkl_intel_lp64.a $(LAdir)/mkl/lib/intel64/libmkl_intel_thread.a $(LAdir)/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -ldl $(HPLlibHybrid) F2CDEFS = -DAdd__ -DF77_INTEGER=int -DStringSunStyle HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc) HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib) HPL_OPTS = -DASYOUGO -DHYBRID HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES) CC = mpiicc CCNOOPT = $(HPL_DEFS) -O0 -w -nocompchk MKLINCDIR = -I"/opt/intel/mkl/include" CCFLAGS = $(HPL_DEFS) $(MKLINCDIR) -O3 -w -ansi-alias -i-static -z noexecstack -z relro -z now -openmp -nocompchk LINKER = $(CC) LINKFLAGS = $(CCFLAGS) -openmp -mt_mpi $(STATICFLAG) -nocompchk