Dear All,

      I am using IntelMPI + ifort + MKL to compile Quantum-Espresso 6.1. Everthing works fine except invoking scalapack routines. Calls to PDPOTRF may exit with non-zero error code under certain circumstance. In an example, with 2 nodes * 8 processors per node the program works but with 4 nodes * 4 processors per node the program fails. If I_MPI_DEBUG is used,  for the failed case there are following messages just before the call exit with code 970, while for the working case there is no such messages:

[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676900, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675640, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x26742b8, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676b58, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x26769c8, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676c20, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675fa0, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676068, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676a90, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676e78, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2678778, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675898, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675a28, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675bb8, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2674f38, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676ce8, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2676130, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2674768, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2674448, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2674b50, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675e10, operation 2, size 12272, lkey 1879682311
[10#18754:18754@node09] MRAILI_Ext_sendq_send(): rail 0,vbuf 0x2675708, operation 2, size 2300, lkey 1879682311

         Could you provide any suggestion about what is the possible cause here? Thanks very much.

Feng

Hello everyone,

I am writing because I am currently implementing a failure recovery system for a cluster with Intel OmniPath that will be designated for handling computations in a physical experiment. What I want to implement is a mechanism to detect a node that failed and to notify rest of the nodes. I tried to check the node failure by invoking psm2_poll. Unfortunately, as I saw in the Intel ® Performance ScaledMessaging 2 (PSM2) Programmer’s Guide, this function does not return errors (values) other than OK or OK_NO_PROGRESS (this is at least what I have observed in my application - the poll on a dead node behaves as if the node did not fail/disconnect and did not send any message). 

So the question is: What are the methods of notifying other nodes after node failure ? Is there a lightweight function that I can invoke along with poll to check if the node from whom I am trying to get messages exists ?

In worst case, I can implement this using a counter and a timeout, but if there is a mechanism supported by the API, I am wide open.

Best Regards

I'm trying to port old (working) PBS scripts to SLURM, but not succeeding.

Can anybody share a working SLURM subscription script for coarray distributed memory please.

The best I can do is:

#!/bin/bash

#SBATCH --nodes=2
#SBATCH --tasks-per-node=28
#SBATCH --time=00:30:00
#SBATCH -p test

EXE=testABW.x
export I_MPI_COLL_INTRANODE=pt2pt
module load intel/2017.01
cd $SLURM_SUBMIT_DIR
echo -genvall -genv -np ${SLURM_NTASKS} ./$EXE > xx14.conf
srun -N ${SLURM_NNODES} --nodelist=${SLURM_NODELIST} /bin/hostname > nodes
./$EXE

which gives me:

Lmod has detected the following error: The following module(s) are unknown:"languages/intel/2017.01"

Please check the spelling or version number. Also try "module spider ..."

Error in system call pthread_mutex_unlock: Operation not permitted
    ../../src/mpid/ch3/channels/nemesis/netmod/tmi/tmi_poll.c:629
Error in system call pthread_mutex_unlock: Operation not permitted
    ../../src/mpid/ch3/channels/nemesis/netmod/tmi/tmi_poll.c:629
Error in system call pthread_mutex_unlock: Operation not permitted
    ../../src/mpid/ch3/channels/nemesis/netmod/tmi/tmi_poll.c:629
Error in system call pthread_mutex_unlock: Operation not permitted
    ../../src/mpid/ch3/channels/nemesis/netmod/tmi/tmi_poll.c:629

Thanks!

For a simple mpi program compiled with Intel compiler studio 2016 and 2017, with intel compiler and mpi, the jobs fail with the following debug errors. The code will run extremely slowly, and get stuck for about 30 seconds at one stage if run on 2 or more nodes. It runs smoothly without any problems on a single node. The same code compiled with gcc and openmpi runs smoothly without any problem on any number of nodes.

Do you know what might be the problem? Thanks.

Hi, I would like to know if Intel Parallel Studio XE 2017 (Cluster edition) and rocks cluster 6.2 are compatibles (without having problems to install). I have a cluster with 1 Head node and 6 nodes. And.. is enought 1 license for all the cluster? (to run things with Intel MPI).

Thanks!

Pablo

This particular problem is likely not an Intel problem, but may be one experienced here by someone, and who has some advice for resolution.

I have a compute intensive application that is written as MPI distributed, OpenMP threaded. I can run this program in my office directly (no MPI) or distributed (mpirun), I can run on 1 or 2 nodes using MPI. The systems locally are Xeon host (Cent OS) and KNL host (Cent OS). I've also have run this successfully by ssh-ing into the Colfax Cluster using 1 to 8 KNL's (couldn't get 16 KNLs to schedule).

I am now running (attempting to run) test on a hardware vendor's setup.

After resolving configuration issues and installation issues I can

ssh into their login server (Xeon host)
su to super user
ssh to KNL node (Xeon KNL)
mpirun ... using two KNL's

When I mpirun'd the application, it started up as expected (periodically emitting progress information to the console). Several minutes into the run it hung. I thought this was a programming error resulting in deadlock, or maybe a watchdog timer killed a thread or process without killing the mpi process manager.

To eliminate possible causes I started the application as stand alone (without mpirun).

Several minutes into this, the program hung as well. So not mpi messaging issue.

Pressing Ctrl-C on the keyboard (through two ssh connections) yielded no response (application not killed). I thought one of the systems in the ssh connections went down. Prior to doing anything on those ssh connections, I wrote an email to my client explaining the hang issue. Several minutes passed.

Now for the interesting part.

After this several minutes "hang" 100 to 150 lines of progress output from the application came out on the console window then the program terminated by Ctrl-C message came out. What appears to have happened was the application was running fine during the console hang, but the terminal output was suspended (as if flow control instructed it to stop). And no, I did not press Ctrl-S or the Pause key.

Anyone have information on this and how to avoid the hang, or at least how to resume the console output without killing the application.

Jim Dempsey

We recently installed a cluster using Rocks 6.2, OFED-3.18-3, and Intel Parallel Studio XE Cluster Edition 2016.4.258.

My parallel code can run and the performance is good, but the code output says

 ERR: corrupt dat.conf entry field: EOR, EOF, file offset=5885
 ERR: corrupt dat.conf entry field: EOR, EOF, file offset=5885
 ERR: corrupt dat.conf entry field: api_ver, file offset=5974
 ERR: corrupt dat.conf entry field: api_ver, file offset=6061
 ERR: corrupt dat.conf entry field: api_ver, file offset=6148
 ERR: corrupt dat.conf entry field: api_ver, file offset=5974
 ERR: corrupt dat.conf entry field: api_ver, file offset=6061
 ERR: corrupt dat.conf entry field: api_ver, file offset=6148
 ERR: corrupt dat.conf entry field: EOR, EOF, file offset=5885

...

I tried to google it, but can not find useful information. Thanks in advance for any help.

Best wishes,
Ding

Hello,

We have some troubles on our cluster to use Intel MPI with PBSPro under a Kerberized environment.

The thing is PBSPro doesn't forward Kerberos tickets which prevents us to have a password-less ssh. Security officers rejects ssh keys without a passphrase, beside, we are expected to rely on Kerberos in order to connect through ssh.

As you can expect, a simple

mpirun -l -v -n $nb_procs "${PBS_O_WORKDIR}/echo-node.sh" # that simply calls bash builtin echo

fails because of pmi_proxy that hangs, and in the end the walltime is exceeded, and we observe:

[...]
[mpiexec@node028.sis.cnes.fr] Launch arguments: /work/logiciels/rhall/intel/parallel_studio_xe_2017_u2/compilers_and_libraries_2017.2.174/linux/mpi/intel64/bin/pmi_proxy --control-port node028.sis.cnes.fr:41735 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --rmk pbs --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1939201911 --usize -2 --proxy-id 0
[mpiexec@node028.sis.cnes.fr] Launch arguments: /bin/ssh -x -q node029.sis.cnes.fr /work/logiciels/rhall/intel/parallel_studio_xe_2017_u2/compilers_and_libraries_2017.2.174/linux/mpi/intel64/bin/pmi_proxy --control-port node028.sis.cnes.fr:41735 --debug --pmi-connect alltoall --pmi-aggregate -s 0 --rmk pbs --launcher ssh --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 1939201911 --usize -2 --proxy-id 1
[proxy:0:0@node028.sis.cnes.fr] Start PMI_proxy 0
[proxy:0:0@node028.sis.cnes.fr] STDIN will be redirected to 1 fd(s): 17
[0] node: 0 /  /
=>> PBS: job killed: walltime 23 exceeded limit 15
[mpiexec@node028.sis.cnes.fr] HYDU_sock_write (../../utils/sock/sock.c:418): write error (Bad file descriptor)
[mpiexec@node028.sis.cnes.fr] HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:252): unable to write data to proxy
[mpiexec@node028.sis.cnes.fr] ui_cmd_cb (../../pm/pmiserv/pmiserv_pmci.c:174): unable to send signal downstream
[mpiexec@node028.sis.cnes.fr] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@node028.sis.cnes.fr] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:501): error waiting for event
[mpiexec@node028.sis.cnes.fr] main (../../ui/mpich/mpiexec.c:1147): process manager error waiting for completion

If instead we log onto the master node, execute kinit, and then run mpirun, everything works fine. Except this isn't exactly an acceptable workaround.

I've tried to play with the fabrics as the nodes are also connected with infiband, but I had no luck there. If I'm not mistaken, pmi_proxy does require password-less ssh whatever fabrics we have. Am I right ?

BTW, I've also tried to play with Altair PBSPro's pbsdsh. I've observed that the parameters it expects are not compatible with the one fed by mpirun. Besides, even if I encapsulate pbsdsh, pmi_proxy still fails with a

[proxy:0:0@node028.sis.cnes.fr] HYDU_sock_connect (../../utils/sock/sock.c:268): unable to connect from "node028.sis.cnes.fr" to "node028.sis.cnes.fr" (Connection refused)
[proxy:0:0@node028.sis.cnes.fr] main (../../pm/pmiserv/pmip.c:461): unable to connect to server node028.sis.cnes.fr at port 49813 (check for firewalls!)

So. My question, is there a workaround? Something that I've missed? Every clue I can gather googling and experimenting points me towards "password-less ssh". So far the only workaround we've found consist in using another MPI framework :(

Regards,

I'm using ifort and mpiifort 17.0.1. Previously I was able to run Fortran coarray programs with PBS. Now I need to use SLURM, and I cannot  adjust my old PBS scripts for this to work. I get lots of errors like this:

Error in system call pthread_mutex_unlock: Operation not permitted
    ../../src/mpid/ch3/channels/nemesis/netmod/tmi/tmi_poll.c:629

My coarray SLURM script is:

#!/bin/bash

#SBATCH --nodes=2
#SBATCH --tasks-per-node=28
#SBATCH --time=00:30:00
#SBATCH -p test

EXE=testABW.x
export I_MPI_COLL_INTRANODE=pt2pt
module load intel/2017.01
cd $SLURM_SUBMIT_DIR
echo -genvall -genv -np ${SLURM_NTASKS} ./$EXE > xx14.conf
srun -N ${SLURM_NNODES} --nodelist=${SLURM_NODELIST} /bin/hostname > nodes
./$EXE

The executable and the Intel coarray distributed memory config file are created according to Intel's instructions:

https://software.intel.com/en-us/articles/distributed-memory-coarray-fortran-with-the-intel-fortran-compiler-for-linux-essential

Again, all that's changed is that I need to use SLURM instead of PBS. The executable hasn't changed.

Thanks

Hello,

I try to start an MPI job on with the following settings.

I have two nodes, workstation1 and workstation2.
I can ssh from workstation1 (10.0.0.1) to workstation2 (10.0.0.') without password. I've already arranged rsa keys.
I can ssh from both workstation1 and workstation2 to themselves without password.
I can ping from 10.0.0.1 to 10.0.0.2 and from 10.0.0.2 to 10.0.0.1

workstation 1 & workstation2 are connected via Mellanox inifiniband.
I'm running Intel(R) MPI Library, Version 2017 Update 2  Build 20170125
I've installed MLNX_OFED_LINUX-4.1-1.0.2.0-ubuntu16.04-x86_64

workstation1 /etc/hosts :

127.0.0.1    localhost
10.0.0.1    workstation1

# The following lines are desirable for IPv6 capable hosts
#::1     ip6-localhost ip6-loopback
#fe00::0 ip6-localnet
#ff00::0 ip6-mcastprefix
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters

# mpi nodes
10.0.0.2 workstation2

-------------------------------------------------------------
workstation2 /etc/hosts :

127.0.0.1    localhost
10.0.0.2    workstation2

# The following lines are desirable for IPv6 capable hosts
#::1     ip6-localhost ip6-loopback
#fe00::0 ip6-localnet
#ff00::0 ip6-mcastprefix
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters

#mpi nodes
10.0.0.1 workstation1

--------------------------------------------------------------
Here's my application start command, (simplified app names and params)

#!/bin/bash
export PATH=$PATH:$PWD:/opt/intel/compilers_and_libraries_2017.2.174/linux/mpi/intel64/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$I_MPI_ROOT/intel64/lib:../program1/bin:../program2/bin
export I_MPI_FABRICS=dapl:dapl
export I_MPI_DEBUG=6
export I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1

# Due to the bug in IntelMPI, -genv I_MPI_ADJUST_BCAST "9" flags has been added.
# Mode detailed information is available : https://software.intel.com/en-us/articles/intel-mpi-library-2017-known-issue-mpi-bcast-hang-on-large-user-defined-datatypes

mpirun -l -genv I_MPI_ADJUST_BCAST "9" -genv I_MPI_PIN_DOMAIN=omp
: -n 1 -host 10.0.0.1 ../program1/bin/program1 master stitching stitching \
: -n 1 -host 10.0.0.2 ../program1/bin/program1 slave dissemination \
: -n 1 -host 10.0.0.1 ../program1/bin/program2 param1 param2

-------------------------------------------

I can start my application in dual node with export I_MPI_FABRICS=tcp:tcp, but when I start with dapl:dapl it gives the following error :

OUTPUT :

0] [0] MPI startup(): Intel(R) MPI Library, Version 2017 Update 2  Build 20170125 (id: 16752)
[0] [0] MPI startup(): Copyright (C) 2003-2017 Intel Corporation.  All rights reserved.
[0] [0] MPI startup(): Multi-threaded optimized library
[0] [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1
[1] [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1
[2] [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1
[0] [0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1
[0] [0] MPI startup(): dapl data transfer mode
[1] [1] MPI startup(): DAPL provider ofa-v2-mlx4_0-1
[2] [2] MPI startup(): DAPL provider ofa-v2-mlx4_0-1
[1] [1] MPI startup(): dapl data transfer mode
[2] [2] MPI startup(): dapl data transfer mode
[0] [0:10.0.0.1] unexpected DAPL event 0x4003
[0] Fatal error in PMPI_Init_thread: Internal MPI error!, error stack:
[0] MPIR_Init_thread(805): fail failed
[0] MPID_Init(1831)......: channel initialization failed
[0] MPIDI_CH3_Init(147)..: fail failed
[0] (unknown)(): Internal MPI error!
[1] [1:10.0.0.2] unexpected DAPL event 0x4003
[1] Fatal error in PMPI_Init_thread: Internal MPI error!, error stack:
[1] MPIR_Init_thread(805): fail failed
[1] MPID_Init(1831)......: channel initialization failed
[1] MPIDI_CH3_Init(147)..: fail failed
[1] (unknown)(): Internal MPI error!

Do you have any idea what could be the cause? By the way, on single node with dapl, I can start my application on both computers separately (meaning -host 10.0.0.1 for all application for workstation1, never attaching 10.0.0.2 related apps).

Hello,

I was running the IMB-MPI pingpong job by using following scripts:

mpirun -hosts node1,node2 -ppn 1 -n 2 -env I_MPI_FABRICS=dapl -env I_MPI_DAPL_PROVIDER=ofa-v2-mlx5_0-1s -env I_MPI_DEBUG=100 -env I_MPI_DYNAMIC_CONNECTION=0 IMB-MPI1 pingpong

the debug info is as follows:

could you please give some suggestions how to fix this problem. thanks in advance.

Hi,

I observed that Intel MPI is now part of the Intel® Performance Libraries, which I can freely download via https://software.intel.com/en-us/performance-libraries. However, following https://software.intel.com/en-us/articles/end-user-license-agreement, unlike the other Performance Libraries,  it is not licensed under the https://software.intel.com/en-us/license/intel-simplified-software-license. I assume in most cases it would be a "Named User" license then, for the user who downloads the software. Furthermore under https://software.intel.com/en-us/articles/free-mkl Intel MPI is not listed under "Community Licensing for Everyone", but it *is* listed under "Use as an Academic Researcher".

However my setting is a little different (analyst at academic HPC centre). My questions are as follows then:

1. Can an academic supercomputer center with a license to Intel Parallel Studio XE Professional/Composer edition (so not the Cluster edition!) still download Intel MPI via https://software.intel.com/en-us/performance-libraries and make it available to its users (without Premier Support obviously).

2. Can an academic supercomputer center without any Intel Parallel Studio XE license still download Intel MPI via https://software.intel.com/en-us/performance-libraries and make it available to its users (using GCC)

3. Can individual academic researchers (so they are registered themselves, instead of the cluster admins) download Intel MPI via https://software.intel.com/en-us/performance-libraries and use it, using Intel Parallel Studio XE Professional/Composer edition installed by cluster admins?

4. Can individual academic researchers (so they are registered themselves, instead of the cluster admins) download Intel MPI via https://software.intel.com/en-us/performance-libraries and use it, using GCC on a cluster?

The CAE software we develop is using Intel MPI version 4.1 and is typically deployed on clusters with infiniband.
One of our customers is considering the purchase of a cluster that uses omni-path.
Can anyone comment on whether or not I should expect the software to work as-is with omni-path?  Or would we need to upgrade our Intel MPI, or perhaps make other code changes?

Thank you,

Eric Marttila
ThermoAnalytics, Inc.
eam@thermoanalytics.com

I am trying to run the HPC orchestrator integration tests. The Fortran tests fails as I do not have a license for the Fortran compiler from Intel - I only have a C++ compiler license. Does anybody know how I could disable the Fortran compiler tests? I did not find anything in the documentation or the test scripts.
 

Hello, I have been trying to submit a job in our cluster for a intel17 compiled and impi enabled code. I keep getting trouble at startup when running through PBS.

This is the submission script:

#!/bin/bash
#PBS -N propane_XO2_ramp_dx_p3125cm(IMPI)
#PBS -W umask=0022
#PBS -e /home4/mnv/FIREMODELS_ISSUES/fds/Validation/UMD_Line_Burner/Test_Valgrind/propane_XO2_ramp_dx_p3125cm.err
#PBS -o /home4/mnv/FIREMODELS_ISSUES/fds/Validation/UMD_Line_Burner/Test_Valgrind/propane_XO2_ramp_dx_p3125cm.log
#PBS -l nodes=16:ppn=12
#PBS -l walltime=999:0:0
module purge
module load null modules torque-maui intel/17
export OMP_NUM_THREADS=1
export I_MPI_FABRICS=shm:dapl
export I_MPI_DAPL_PROVIDER=OpenIB-cma
export I_MPI_FALLBACK_DEVICE=0
export I_MPI_DEBUG=100
cd /home4/mnv/FIREMODELS_ISSUES/fds/Validation/UMD_Line_Burner/Test_Valgrind
echo
echo $PBS_O_HOME
echo `date`
echo "Input file: propane_XO2_ramp_dx_p3125cm.fds"
echo " Directory: `pwd`"
echo "      Host: `hostname`"
/opt/intel17/compilers_and_libraries/linux/mpi/bin64/mpiexec   -np 184 /home4/mnv/FIREMODELS_ISSUES/fds/Build/impi_intel_linux_64/fds_impi_intel_linux_64 propane_XO2_ramp_dx_p3125cm.fds

As you can see I'm invoking DAPL and OpenIB-cma as dapl provider. This is what I see on my login node /etc/dat.conf

OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib0 0"""
OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib1 0"""
OpenIB-cma-2 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib2 0"""
OpenIB-cma-3 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib3 0"""
OpenIB-bond u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "bond0 0"""
ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib0 0"""
ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib1 0"""
ofa-v2-ib2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib2 0"""
ofa-v2-ib3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib3 0"""
ofa-v2-bond u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "bond0 0"""

Now logging in to the actual compute nodes I don't see an /etc/dat.conf on these. I don't know if this is normal or there is an issue there.

Anyways, when I submit the job I get the following attached stdout file, where it seems some of the nodes fail to load OpenIB-cma (with no fallback fabrics).

To be sure, some nodes on the cluster use Qlogic infiniband cards and others use Mellanox.

At this point I've tried several combinations, either specifying or not ib fabrics, without success. I'd really appreciate if you help me troubleshooting this.

Thank you,

Marcos

Hey, good morning. My name is Eliomar, im from venezuela. im a bit new working with MPI, im making a masters project with this technology. But im facing a problem with my implementation that i dont know how to face it.

The problem is, i have a cluster, composed of 4 skylakes and 1 knl. im trying to run a program in knl from skylake. In knl i have installed the 2017 version and in skylakes i have 2015 version. in the beggining it crashes with a bash error saying that the file or directory does not exist. thats correct because i dont have same versions on the knl and when i was doing MPI_Comm_spawn to the knl it should be right that error. i thought i solved that error setting the root environmental variables, but then when i run mi program in the cluster at the moment of spawn a new process but now in the knl. the programs just hangs there. the errors that it says are: (after i push ctrl+c)

HYDU_sock_write (../../utils/sock/sock.c:417): write error (Bad file descriptor)

HYD_pmcd_pmiserv_send_signal (../../pm/pmiserv/pmiserv_cb.c:246): unable to write data to proxy

ui_cmd_cb (../../pm/pmiserv/pmiserv_pmci.c:172): unable to send signal downstream

HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status

HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:480): error waiting for event

main (../../ui/mpich/mpiexec.c:945): process manager error waiting for completion

Any help would be great. if you have a related problem with mine.

Regards from venezuela.

Hi all,

My understanding (correct me if I'm wrong), is that MPI_Mprobe() has to guarantee progress if a matching send has been posted. The minimal working example below, however, runs to completion on a single Phi node of stampede2, while deadlocking on more than one node.

Thanks,
Toby

impi version:
Intel(R) MPI Library for Linux* OS, Version 2017 Update 3 Build 20170405 (id: 17193)

mwe.c (attached)

slurm-mwe-stampede2-two-nodes.sh
~~~
#!/bin/sh
#SBATCH -J mwe # Job name
#SBATCH -p development # Queue (development or normal)
#SBATCH -N 2 # Number of nodes
#SBATCH --tasks-per-node 1 # Number of tasks per node
#SBATCH -t 00:01:00 # Time limit hrs:min:sec
#SBATCH -o mwe-%j.out # Standard output and error log
~~~

mwe-341107.out
~~~
TACC: Starting up job 341107
TACC: Starting parallel tasks...
[0]: post Isend
[1]: post Isend
slurmstepd: error: *** JOB 341107 ON c455-084 CANCELLED AT 2017-10-16T10:59:26 DUE TO TIME LIMIT ***
[mpiexec@c455-084.stampede2.tacc.utexas.edu] control_cb (../../pm/pmiserv/pmiserv_cb.c:857): assert (!closed) failed
[mpiexec@c455-084.stampede2.tacc.utexas.edu] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@c455-084.stampede2.tacc.utexas.edu] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:501): error waiting for event
[mpiexec@c455-084.stampede2.tacc.utexas.edu] main (../../ui/mpich/mpiexec.c:1147): process manager error waiting for completion
~~~

Hello, currently I am new to using IMB benchmark and wanted to make sure 

whether checking the configured number of processes (np) is evenly allocated in the nodes and each cores 

Currently the IMB-benchmark provides the throughput per sec based on the fixed size of message with fixed size of duration 

For example like this one below. 

$ mpirun -np 64 -machinefile hosts_infin ./IMB-MPI1 -map 32x2 Sendrecv

#-----------------------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec
            0         1000         0.76         0.76         0.76         0.00
            1         1000         0.85         0.85         0.85         2.35
            2         1000         0.79         0.79         0.79         5.06
            4         1000         0.80         0.80         0.80        10.00
            8         1000         0.78         0.78         0.78        20.45
           16         1000         0.79         0.80         0.80        40.16
           32         1000         0.79         0.79         0.79        80.61
           64         1000         0.79         0.79         0.79       162.59
          128         1000         0.82         0.82         0.82       311.41
          256         1000         0.91         0.91         0.91       565.42
          512         1000         0.95         0.95         0.95      1082.13
         1024         1000         0.99         0.99         0.99      2076.87
         2048         1000         1.27         1.27         1.27      3229.91
         4096         1000         1.71         1.71         1.71      4802.87
         8192         1000         2.49         2.50         2.50      6565.97
        16384         1000         4.01         4.01         4.01      8167.28
        32768         1000         7.08         7.09         7.08      9249.23
        65536          640        22.89        22.89        22.89      5725.50
       131072          320        37.45        37.45        37.45      6999.22
       262144          160        65.74        65.76        65.75      7972.53
       524288           80       120.10       120.15       120.12      8727.37
      1048576           40       228.63       228.73       228.68      9168.57
      2097152           20       445.38       445.69       445.53      9410.86
      4194304           10       903.77       905.97       904.87      9259.29

#-----------------------------------------------------------------------------

However, this does not guarantee that the processes are evenly distributed to the nodes  when I configure the different  varying number of processors.

Would assigning to the specific core be possible in IMB-benchmark? or do I need to use the alternative benchmark to do this? 

The Intel® HPC Developer Conference, taking place November 11–12, 2017 in Denver, Colorado, is a great opportunity for developers who want to gain technical knowledge and hands-on experience with HPC luminaries to help take full advantage of today’s and tomorrow’s technology. Topics this year will include parallel programming, high productivity languages, artificial intelligence, systems, enterprise, visualization development... and much more.

We invite you to come gain hands-on experience with Intel platforms, network with Intel and industry experts, and gain insights on recent technology advances to maximize software efficiency and accelerate your path to discovery. Register now for this free conference.*

Encourage your network of colleagues to attend too.  When leveraging social media use hashtag #HPCDevCon.

Hi There,

I have a system with 6 computenodes, /opt folder is nfs shared and intel parallel studio cluster version installed on nfs server.

I am using slurm as workload manager. When i run a vasp job on 1 node there is no problem, But when i start to run the job on 2 or more nodes i am getting the following errors;

rank = 28, revents = 29, state = 1
Assertion failed in file ../../src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 2988: (it_plfd->revents & POLLERR) == 0
internal ABORT - process 0

I tested the ssh between computenodes with sshconnectivity.exp /nodefile

The user information is shared over ldap server which is headnode.

I couldn't find a working solution in the net. Do anyone has ever had this error?

Thanks.