anirudh2290 opened a new issue #9997: Flaky test: test_sparse_embedding URL: https://github.com/apache/incubator-mxnet/issues/9997 ## Description Please see: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-9869/13/pipeline The test_sparse_embedding fails in assert_almost_equal. At first look, it feels like setting specific tolerance values should fix the issue. Even in the failed example, the values of a and b at the point of error are the same. ``` ====================================================================== FAIL: test sparse embedding operator ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc return func(*arg, **kw) File "/workspace/tests/python/gpu/../unittest/common.py", line 157, in test_new orig_test(*args, **kwargs) File "/workspace/tests/python/gpu/../unittest/test_sparse_operator.py", line 1675, in test_sparse_embedding check_sparse_embedding(in_dim, out_dim, batch, densities, False) File "/workspace/tests/python/gpu/../unittest/test_sparse_operator.py", line 1668, in check_sparse_embedding assert_almost_equal(grad_map["embed_weight"].asnumpy(), np.dot(np_onehot.T, grad.asnumpy())) File "/workspace/python/mxnet/test_utils.py", line 493, in assert_almost_equal raise AssertionError(msg) AssertionError: Items are not equal: Error 7.891414 exceeds tolerance rtol=0.000010, atol=0.000000. Location of maximum error:(24, 1), a=0.000378, b=0.000378 a: array([[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ],... b: array([[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ],... -------------------- >> begin captured logging << -------------------- common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1475581309 to reproduce. --------------------- >> end captured logging << --------------------- ``` ## Environment info (Required) Package used (Python/R/Scala/Julia): Python ## Build info (Required if built from source) Compiler (gcc/clang/mingw/visual studio):gcc MXNet commit hash: 02dd89a68f659c2a9b0bff62c54c50dff1151f6b ## Error Message: ``` AssertionError: Items are not equal: Error 7.891414 exceeds tolerance rtol=0.000010, atol=0.000000. Location of maximum error:(24, 1), a=0.000378, b=0.000378 a: array([[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ],... b: array([[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ],... ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
