chrishkchris opened a new issue #744: URL: https://github.com/apache/singa/issues/744
The default of atot for onnx "model" is here https://github.com/onnx/onnx/blob/master/onnx/backend/test/case/model/__init__.py#L34 The default of atot for onnx "node" is here https://github.com/onnx/onnx/blob/master/onnx/backend/test/case/node/__init__.py#L148 Is that possible to override these setting in our test? Currently when run our onnx_test_backend.py, it all reports "Not equal to tolerance rtol=0.001, atol=1e-07" I think absolute value error atol=1e-07 is too tough, it is too difficult to pass? ``` root@d05828f767ee:~/dcsysh/singa/test/python# python3 test_onnx_backend.py ss............................ssssssssssssssssssssssssssssssss................ssss..ss..ss......FFFF..ssssssssss..ssssssssssssssss............ssss....................ssssssss........................ssss....ssssssssssssssssssssssssss........ssssssssssssssssssssssssssss..........ssssssssssssss......FsFsssssssssssssssss..................ssss....ssssss..ssss..........ss.s............ssss....ssssssssssssssssssss........ssssssssssss..............ssssssssssssssssssssssss......ss......ssss..ss........FFFF..ssssssssss............ssssssssssssssssssssssssss......ssss....ssssssssssssssssss..................................ss........ssssssssssssssss....ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss................ssssssssssssssssssssssssssssssss................ssssssssssssssssss....................ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..........ssss......ssss........ssss..............ss..........................................ssssssssssss....................ssssssssssssssssssss....ssssssssssssssssssssssssssssss................ssssss................ ====================================================================== FAIL: test_averagepool_2d_same_lower_cpu (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 189 / 3072 (6.15%) Max absolute difference: 1.3230393 Max relative difference: 3. x: array([[[[ 1.764052e+00, 1.082105e+00, 6.894476e-01, ..., 1.501069e+00, 8.121531e-01, 2.665550e-01], [ 4.381333e-01, -1.760931e-01, -2.374533e-01, ...,... y: array([[[[ 4.410131e-01, 5.410524e-01, 3.447238e-01, ..., 7.505345e-01, 4.060766e-01, 1.332775e-01], [ 2.190667e-01, -1.760931e-01, -2.374533e-01, ...,... ====================================================================== FAIL: test_averagepool_2d_same_lower_cuda (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 189 / 3072 (6.15%) Max absolute difference: 1.3230393 Max relative difference: 3. x: array([[[[ 1.764052e+00, 1.082105e+00, 6.894476e-01, ..., 1.501069e+00, 8.121531e-01, 2.665550e-01], [ 4.381333e-01, -1.760931e-01, -2.374533e-01, ...,... y: array([[[[ 4.410131e-01, 5.410524e-01, 3.447238e-01, ..., 7.505345e-01, 4.060766e-01, 1.332775e-01], [ 2.190667e-01, -1.760931e-01, -2.374533e-01, ...,... ====================================================================== FAIL: test_averagepool_2d_same_upper_cpu (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 189 / 3072 (6.15%) Max absolute difference: 0.9547153 Max relative difference: 3. x: array([[[[-0.176093, -0.237453, 0.757017, ..., 0.112902, -0.50158 , -0.67406 ], [-0.773234, -1.090172, -0.339745, ..., 0.040076, -0.369122,... y: array([[[[-0.176093, -0.237453, 0.757017, ..., 0.112902, -0.50158 , -0.33703 ], [-0.773234, -1.090172, -0.339745, ..., 0.040076, -0.369122,... ====================================================================== FAIL: test_averagepool_2d_same_upper_cuda (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 189 / 3072 (6.15%) Max absolute difference: 0.9547153 Max relative difference: 3. x: array([[[[-0.176093, -0.237453, 0.757017, ..., 0.112902, -0.50158 , -0.67406 ], [-0.773234, -1.090172, -0.339745, ..., 0.040076, -0.369122,... y: array([[[[-0.176093, -0.237453, 0.757017, ..., 0.112902, -0.50158 , -0.33703 ], [-0.773234, -1.090172, -0.339745, ..., 0.040076, -0.369122,... ====================================================================== FAIL: test_equal_bcast_cpu (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 1 / 60 (1.67%) x: array([[[False, False, False, False, False], [False, False, False, False, False], [False, False, False, False, False],... y: array([[[False, False, False, False, False], [False, False, False, False, False], [False, False, False, False, False],... ====================================================================== FAIL: test_equal_cpu (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 1 / 60 (1.67%) x: array([[[False, False, False, False, False], [False, False, False, False, False], [False, False, False, False, True],... y: array([[[False, False, False, False, False], [False, False, False, False, False], [False, False, False, False, True],... ====================================================================== FAIL: test_maxpool_2d_same_lower_cpu (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 35 / 3072 (1.14%) Max absolute difference: 1.6961312 Max relative difference: 0. x: array([[[[ 1.764052e+00, 1.764052e+00, 9.787380e-01, ..., 1.532779e+00, 1.469359e+00, 3.781625e-01], [ 1.764052e+00, 1.764052e+00, 9.787380e-01, ...,... y: array([[[[ 1.764052e+00, 1.764052e+00, 9.787380e-01, ..., 1.532779e+00, 1.469359e+00, 3.781625e-01], [ 1.764052e+00, 1.764052e+00, 9.787380e-01, ...,... ====================================================================== FAIL: test_maxpool_2d_same_lower_cuda (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 35 / 3072 (1.14%) Max absolute difference: 1.6961312 Max relative difference: 0. x: array([[[[ 1.764052e+00, 1.764052e+00, 9.787380e-01, ..., 1.532779e+00, 1.469359e+00, 3.781625e-01], [ 1.764052e+00, 1.764052e+00, 9.787380e-01, ...,... y: array([[[[ 1.764052e+00, 1.764052e+00, 9.787380e-01, ..., 1.532779e+00, 1.469359e+00, 3.781625e-01], [ 1.764052e+00, 1.764052e+00, 9.787380e-01, ...,... ====================================================================== FAIL: test_maxpool_2d_same_upper_cpu (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 37 / 3072 (1.2%) Max absolute difference: 1.2028884 Max relative difference: 0. x: array([[[[ 1.764052, 0.978738, 2.240893, ..., 1.469359, 0.378163, 0.378163], [ 0.177426, -0.347912, 0.462782, ..., 0.976639, 0.706573,... y: array([[[[ 1.764052, 0.978738, 2.240893, ..., 1.469359, 0.378163, 0.378163], [ 0.177426, -0.347912, 0.462782, ..., 0.976639, 0.706573,... ====================================================================== FAIL: test_maxpool_2d_same_upper_cuda (__main__.OnnxBackendNodeModelTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func return test_func(*args, device=device, **kwargs) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run atol=model_test.atol) File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs atol=atol) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=0.001, atol=1e-07 Mismatched elements: 37 / 3072 (1.2%) Max absolute difference: 1.2028884 Max relative difference: 0. x: array([[[[ 1.764052, 0.978738, 2.240893, ..., 1.469359, 0.378163, 0.378163], [ 0.177426, -0.347912, 0.462782, ..., 0.976639, 0.706573,... y: array([[[[ 1.764052, 0.978738, 2.240893, ..., 1.469359, 0.378163, 0.378163], [ 0.177426, -0.347912, 0.462782, ..., 0.976639, 0.706573,... ---------------------------------------------------------------------- Ran 1114 tests in 2.126s ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
