Hello,
we are trying to speed up a parallel program using the Intel Compiler and the OpenMP library.
We have observed that the program hangs after running ok for 3-4 days, in one of the parallel loops. The binary keeps running but stays in a permanent waiting state. Here is the gdb stack trace:
#0 0x00007fefbc1bf705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fefbc98e9ce in __kmp_suspend_template (th_gtid=, flag=) at ../../src/z_Linux_util.c:1819
#2 __kmp_suspend_64 (th_gtid=-1145176444, flag=0x80) at ../../src/z_Linux_util.c:1874
#3 0x00007fefbc92fe08 in suspend (this=, th_gtid=) at ../../src/kmp_wait_release.h:405
#4 __kmp_wait_template (this_thr=, flag=, final_spin=, itt_sync_obj=) at ../../src/kmp_wait_release.h:224
#5 wait (this=, this_thr=, final_spin=, itt_sync_obj=) at ../../src/kmp_wait_release.h:414
#6 __kmp_hyper_barrier_gather (bt=3149790852, this_thr=0x80, gtid=1, tid=-1, reduce=0x7fefbbbdfe00, itt_sync_obj=0x0) at ../../src/kmp_barrier.cpp:510
#7 0x00007fefbc9331c3 in __kmp_join_barrier (gtid=-1145176444) at ../../src/kmp_barrier.cpp:1364
#8 0x00007fefbc959ee2 in __kmp_internal_join (id=0x7fefbbbdfe84, gtid=128, team=0x1) at ../../src/kmp_runtime.c:7142
#9 0x00007fefbc9609a4 in __kmp_join_call (loc=0x7fefbbbdfe84, gtid=128, exit_teams=1) at ../../src/kmp_runtime.c:2322
#10 0x00007fefbc9345bd in __kmpc_fork_call (loc=0x7fefbbbdfe84, argc=128, microtask=0x1) at ../../src/kmp_csupport.c:326
#11 0x000000000045f4fb in CovarianceMatrixCxx::kalman (......) at G2/CovarianceMatrixCxx.cpp:295
So far, we have observed this problem using Scientfic Linux 7 (glibc 2.17), but not using Scientific Linux 6 (glibc 2.12), when using "icc -openmp". The behaviour is the same when using "icc -fopenmp".
When we use the GNU C++ Compiler with OpenMP (g++ -fopenmp), the program runs fine in both SL6 and SL7.
In summary, this looks like a problem when combining the Intel Compiler and SL7, when the OpenMP library is used. This is the ICC version we are using:
> icc --version
icc (ICC) 15.0.1 20141023
Unfortunately, the program is quite complex and we have not been able to generate a simplified version of the problem that can be easily reproduced. Below is an snapshoot of the loop that is hanging.
Are you aware of any kind of problem similar to this one?
Any help is appreciated,
Javier
#pragma omp parallel for private(i,j,k)
for(i=0; i
{
v_temp[i] = 0.0;
for(k=0; k
j = index_OfNonZero[k];
if(j >= i) {
v_temp[i] += v_Matrix[i][j]*v_KalmanVec[j].d_A;
} else {
v_temp[i] += v_Matrix[j][i]*v_KalmanVec[j].d_A;
}
}//for k
}//for i