rahul003 commented on issue #8915: NVLink communication pattern updated URL: https://github.com/apache/incubator-mxnet/pull/8915#issuecomment-360869841 No, it is still failing ``` test_kvstore_gpu.test_rsp_push_pull ... terminate called after throwing an instance of 'dmlc::Error' what(): [23:44:37] src/engine/./threaded_engine.h:359: [23:44:37] src/ndarray/ndarray_function.cc:181: ElementwiseSum<cpu> has not been implemented for storage_type = << 0 Stack trace returned 10 entries: [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x5a) [0x7f9f39b7334a] [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f9f39b73ee8] [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(void mxnet::ndarray::ElementwiseSum<mshadow::cpu>(mshadow::Stream<mshadow::cpu>*, mxnet::Resource const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, mxnet::NDArray*)+0x6a) [0x7f9f3c16f8aa] [bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(+0x2f0ad9d) [0x7f9f3c1a5d9d] [bt] (4) /workspace/python/mxnet/../../lib/libmxnet.so(+0x330848b) [0x7f9f3c5a348b] [bt] (5) /workspace/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x100) [0x7f9f3c5afb50] [bt] (6) /workspace/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>&&)+0xe2) [0x7f9f3c5b7b42] [bt] (7) /workspace/python/mxnet/../../lib/libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)> (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)> >::_M_run()+0x4a) [0x7f9f3c5b210a] [bt] (8) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f9f45601c80] [bt] (9) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f9f4cf136ba] A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging. Stack trace returned 8 entries: [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x5a) [0x7f9f39b7334a] [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f9f39b73ee8] [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x39a) [0x7f9f3c5afdea] [bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>&&)+0xe2) [0x7f9f3c5b7b42] [bt] (4) /workspace/python/mxnet/../../lib/libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)> (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)> >::_M_run()+0x4a) [0x7f9f3c5b210a] [bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f9f45601c80] [bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f9f4cf136ba] [bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9f4cc493dd] MKL Build:20171227 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
