[GitHub] larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet compiled with DEBUG=1

2019-01-18 Thread GitBox 
larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet 
compiled with DEBUG=1
URL: 
https://github.com/apache/incubator-mxnet/issues/10856#issuecomment-455633028
 
 
   What's worse is that when executing this test, you can easily create a 
segmentation fault, by hitting ^C so we are UB land:
   
   ```
   python tests/python/unittest/test_gluon_data.py
   ```
   
   
   ```
   OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
   OMP: Hint: Please submit a bug report with this message, compile and run 
commands used, and machine configuration info including native compiler and 
operating system versions. Faster response will be obtained by including all 
program sources. For information on submitting this issue, please see 
https://bugs.llvm.org/.
   Assertion failure at kmp_runtime.cpp(6479): __kmp_thread_pool == __null.
   OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
   OMP: Hint: Please submit a bug report with this message, compile and run 
commands used, and machine configuration info including native compiler and 
operating system versions. Faster response will be obtained by including all 
program sources. For information on submitting this issue, please see 
https://bugs.llvm.org/.
   Assertion failure at kmp_runtime.cpp(6479): __kmp_thread_pool == __null.
   OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
   OMP: Hint: Please submit a bug report with this message, compile and run 
commands used, and machine configuration info including native compiler and 
operating system versions. Faster response will be obtained by including all 
program sources. For information on submitting this issue, please see 
https://bugs.llvm.org/.
   ^C[INFO] Setting test np/mx/python random seeds, use 
MXNET_TEST_SEED=1709269671 to reproduce.
   
   --
   Ran 3 tests in 12.296s
   
   OK
   
   Segmentation fault: 11
   
   Stack trace returned 10 entries:
   [bt] (0) 
/home/piotr/devel/mxnet/mxnet/python/mxnet/../../build/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x54)
 [0x7fef667f38e5]
   [bt] (1) 
/home/piotr/devel/mxnet/mxnet/python/mxnet/../../build/libmxnet.so(+0x4ba60be) 
[0x7fef69ba40be]
   [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fefbeb324b0]
   [bt] (3) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0xc7c0d)
 [0x7fefb8857c0d]
   [bt] (4) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0xc7d02)
 [0x7fefb8857d02]
   [bt] (5) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x6ebca)
 [0x7fefb87febca]
   [bt] (6) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x56e9d)
 [0x7fefb87e6e9d]
   [bt] (7) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x5727c)
 [0x7fefb87e727c]
   [bt] (8) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x576b4)
 [0x7fefb87e76b4]
   [bt] (9) 
/home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x56d4f)
 [0x7fefb87e6d4f]
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet compiled with DEBUG=1

2019-01-18 Thread GitBox 
larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet 
compiled with DEBUG=1
URL: 
https://github.com/apache/incubator-mxnet/issues/10856#issuecomment-455632496
 
 
   I spent some time looking at this, I can say I understand what's the 
problem, this happens when the omp engine is initialized after destructors are 
called on thread creation. Usually involves the pthread_at_fork fiasco in 
initialize.cc 
   
   Iinitalization is triggered from __nptl_deallocate_tsd (); 
https://code.woboq.org/userspace/glibc/nptl/pthread_create.c.html#497 
   
   __kmp_team_pool is a volatile which gets changed continously across threads 
for reuse of the last kmp team.
   
   This happens if omp is initialized twice for some reason, so __kmp_team_pool 
is not NULL because there's a thread running omp functions, kmp_team_pool can 
be non-NULL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet compiled with DEBUG=1

2019-01-12 Thread GitBox 
larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet 
compiled with DEBUG=1
URL: 
https://github.com/apache/incubator-mxnet/issues/10856#issuecomment-453742444
 
 
   Stuck for 46h running unit tests with the following error message:
   
   ```
   OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
   OMP: Hint: Please submit a bug report with this message, compile and run 
commands used, and machine configuration info including native compiler and 
operating system versions. Faster response will be obtained by including all 
program sources. For information on submitting this issue, please see 
https://bugs.llvm.org/.
   Assertion failure at kmp_runtime.cpp(6479): __kmp_thread_pool == __null.
   ```
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet compiled with DEBUG=1

2018-11-06 Thread GitBox 
larroy commented on issue #10856: Failed OpenMP assertion when loading MXNet 
compiled with DEBUG=1
URL: 
https://github.com/apache/incubator-mxnet/issues/10856#issuecomment-436361689
 
 
   This is really problematic, is there a suggestion for anything that we can 
do to fix this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services