mseth10 opened a new issue #20289: URL: https://github.com/apache/incubator-mxnet/issues/20289
## Description When enabling CD pipeline for AArch64 https://github.com/apache/incubator-mxnet/pull/20288, observed that 4 unit tests fail/error out. Skipping these tests for now in the CD pipeline and tracking them in this issue. The CD pipeline currently uses python3.6 (observed some more failures with python3.7) ### Error Message Following are the error logs from test CD pipeline: ``` [2021-05-19T22:59:56.273Z] ====================================================================== [2021-05-19T22:59:56.273Z] ERROR: test_contrib_operator.test_multibox_target_op [2021-05-19T22:59:56.273Z] ---------------------------------------------------------------------- [2021-05-19T22:59:56.273Z] Traceback (most recent call last): [2021-05-19T22:59:56.273Z] File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest [2021-05-19T22:59:56.273Z] self.test(*self.arg) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/test_contrib_operator.py", line 285, in test_multibox_target_op [2021-05-19T22:59:56.273Z] assert_allclose(loc_target.asnumpy(), expected_loc_target, rtol=1e-5, atol=1e-5) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/ndarray/ndarray.py", line 2571, in asnumpy [2021-05-19T22:59:56.273Z] ctypes.c_size_t(data.size))) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/base.py", line 246, in check_call [2021-05-19T22:59:56.273Z] raise get_last_ffi_error() [2021-05-19T22:59:56.273Z] mxnet.base.MXNetError: Traceback (most recent call last): [2021-05-19T22:59:56.273Z] [bt] (9) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xd148c) [0xffffadbf448c] [2021-05-19T22:59:56.273Z] [bt] (8) /work/mxnet/python/mxnet/../../lib/libmxnet.so(std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)>, std::shared_ptr<dmlc::ManualEvent> > > >::_M_run()+0x48) [0xffff0cb5e808] [2021-05-19T22:59:56.273Z] [bt] (7) /work/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0x114) [0xffff0cb60084] [2021-05-19T22:59:56.273Z] [bt] (6) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x11c) [0xffff0cb5f3bc] [2021-05-19T22:59:56.273Z] [bt] (5) /work/mxnet/python/mxnet/../../lib/libmxnet.so(+0x88c710) [0xffff0cb54710] [2021-05-19T22:59:56.273Z] [bt] (4) /work/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (mxnet::RunContext), mxnet::imperative::PushOperator(mxnet::OpStatePtr const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, mxnet::DispatchMode)::{lambda(mxnet::RunContext)#4}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&)+0x44) [0xffff0cc17fb4] [2021-05-19T22:59:56.273Z] [bt] (3) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::imperative::PushOperator(mxnet::OpStatePtr const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, mxnet::DispatchMode)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#3}::operator()(mxnet::RunContext, mxnet::engine::CallbackOnComplete) const+0x28c) [0xffff0cc17c5c] [2021-05-19T22:59:56.273Z] [bt] (2) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::op::MultiBoxTargetOp<mshadow::cpu, float>::Forward(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x18a4) [0xffff0d0e0434] [2021-05-19T22:59:56.273Z] [bt] (1) /work/mxnet/python/mxnet/../../lib/libmxnet.so(void mshadow::MultiBoxTargetForward<float>(mshadow::Tensor<mshadow::cpu, 2, float> const&, mshadow::Tensor<mshadow::cpu, 2, float> const&, mshadow::Tensor<mshadow::cpu, 2, float> const&, mshadow::Tensor<mshadow::cpu, 2, float> const&, mshadow::Tensor<mshadow::cpu, 3, float> const&, mshadow::Tensor<mshadow::cpu, 3, float> const&, mshadow::Tensor<mshadow::cpu, 4, float> const&, float, float, float, float, int, mxnet::Tuple<float> const&)+0x11a0) [0xffff0d0d0330] [2021-05-19T22:59:56.273Z] [bt] (0) /work/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x74) [0xffff0c9f8364] [2021-05-19T22:59:56.273Z] File "../src/operator/contrib/multibox_target.cc", line 235 [2021-05-19T22:59:56.273Z] MXNetError: Check failed: temp.size() >= num_negative (0 vs. 1) : ``` ``` [2021-05-19T22:59:56.273Z] ====================================================================== [2021-05-19T22:59:56.273Z] ERROR: test_numpy_interoperability.test_np_array_function_protocol [2021-05-19T22:59:56.273Z] ---------------------------------------------------------------------- [2021-05-19T22:59:56.273Z] Traceback (most recent call last): [2021-05-19T22:59:56.273Z] File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest [2021-05-19T22:59:56.273Z] self.test(*self.arg) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/common.py", line 218, in test_new [2021-05-19T22:59:56.273Z] orig_test(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/util.py", line 297, in _with_np_shape [2021-05-19T22:59:56.273Z] return func(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/util.py", line 481, in _with_np_array [2021-05-19T22:59:56.273Z] return func(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/numpy_dispatch_protocol.py", line 59, in _run_with_array_func_proto [2021-05-19T22:59:56.273Z] .format(func.__name__, str(e))) [2021-05-19T22:59:56.273Z] RuntimeError: Running function test_np_array_function_protocol with NumPy array function protocol failed with exception arrays used as indices must be of integer (or boolean) type [2021-05-19T22:59:56.273Z] common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=382051259 to reproduce. [2021-05-19T22:59:56.273Z] common: WARNING: Error seen with seeded test, use MXNET_TEST_SEED=1596024483 to reproduce. [2021-05-19T22:59:56.273Z] --------------------- >> end captured logging << --------------------- ``` ``` [2021-05-19T22:59:56.273Z] ====================================================================== [2021-05-19T22:59:56.273Z] ERROR: test_numpy_op.test_np_delete [2021-05-19T22:59:56.273Z] ---------------------------------------------------------------------- [2021-05-19T22:59:56.273Z] Traceback (most recent call last): [2021-05-19T22:59:56.273Z] File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest [2021-05-19T22:59:56.273Z] self.test(*self.arg) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/common.py", line 218, in test_new [2021-05-19T22:59:56.273Z] orig_test(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/util.py", line 297, in _with_np_shape [2021-05-19T22:59:56.273Z] return func(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/python/mxnet/util.py", line 481, in _with_np_array [2021-05-19T22:59:56.273Z] return func(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/test_numpy_op.py", line 3636, in test_np_delete [2021-05-19T22:59:56.273Z] expected_ret = _np.delete(a.asnumpy(), obj_onp, axis=axis) [2021-05-19T22:59:56.273Z] File "<__array_function__ internals>", line 6, in delete [2021-05-19T22:59:56.273Z] File "/usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py", line 4406, in delete [2021-05-19T22:59:56.273Z] keep[obj,] = False [2021-05-19T22:59:56.273Z] IndexError: index 0 is out of bounds for axis 0 with size 0 [2021-05-19T22:59:56.273Z] -------------------- >> begin captured logging << -------------------- [2021-05-19T22:59:56.273Z] common: WARNING: Error seen with seeded test, use MXNET_TEST_SEED=39792435 to reproduce. [2021-05-19T22:59:56.273Z] --------------------- >> end captured logging << ---------------------``` ``` ``` [2021-05-19T22:59:56.273Z] ====================================================================== [2021-05-19T22:59:56.273Z] FAIL: test_ndarray.test_ndarray_fluent [2021-05-19T22:59:56.273Z] ---------------------------------------------------------------------- [2021-05-19T22:59:56.273Z] Traceback (most recent call last): [2021-05-19T22:59:56.273Z] File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest [2021-05-19T22:59:56.273Z] self.test(*self.arg) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/common.py", line 218, in test_new [2021-05-19T22:59:56.273Z] orig_test(*args, **kwargs) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/test_ndarray.py", line 1313, in test_ndarray_fluent [2021-05-19T22:59:56.273Z] check_fluent_regular(func, {}) [2021-05-19T22:59:56.273Z] File "/work/mxnet/tests/python/unittest/test_ndarray.py", line 1308, in check_fluent_regular [2021-05-19T22:59:56.273Z] assert almost_equal(regular.asnumpy(), fluent.asnumpy(), equal_nan=equal_nan) [2021-05-19T22:59:56.273Z] AssertionError: [2021-05-19T22:59:56.273Z] -------------------- >> begin captured logging << -------------------- [2021-05-19T22:59:56.273Z] common: WARNING: Error seen with seeded test, use MXNET_TEST_SEED=690006707 to reproduce. [2021-05-19T22:59:56.273Z] --------------------- >> end captured logging << --------------------- ``` ## To Reproduce https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results ## What have you tried to solve it? 1. These errors don't seem to be build related and might be due to python version, so I tried running them with different python versions, but got even more errors with Python 3.7 https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fsrivrohi-release-job-v1.x/detail/srivrohi-release-job-v1.x/72/pipeline/82 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org For additional commands, e-mail: issues-h...@mxnet.apache.org
