leezu edited a comment on pull request #18445: URL: https://github.com/apache/incubator-mxnet/pull/18445#issuecomment-656233768
@ma-hei looking at the output of unix-cpu, I also see that the dataloader related tests are hanging in Tests / Python3: MKLDNN-MKL-CPU. The recordio test you point out (which also hangs) is also part of the data api. cc @zhreshold there may be incompatibility of the data code with Python 3.8 ``` [2020-07-02T19:41:12.626Z] tests/python/unittest/test_gluon_data.py::test_recordimage_dataset_handle PASSED [ 3%] [2020-07-02T19:41:14.506Z] tests/python/unittest/test_gluon_data.py::test_recordimage_dataset_with_data_loader_multiworker PASSED [ 3%] [2020-07-02T20:01:21.465Z] Timeout (0:20:00)! [2020-07-02T20:01:21.465Z] Thread 0x00007fb1f264c700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 379 in _recv [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 414 in _recv_bytes [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 250 in recv [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 576 in _handle_results [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb1f6e4d700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 528 in _handle_tasks [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb1f964e700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/selectors.py", line 415 in select [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 930 in wait [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 499 in _wait_for_updates [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 519 in _handle_workers [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb1f9e4f700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 379 in _recv [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 414 in _recv_bytes [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 250 in recv [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 576 in _handle_results [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb1fc650700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 528 in _handle_tasks [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb200e51700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/selectors.py", line 415 in select [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 930 in wait [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 499 in _wait_for_updates [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 519 in _handle_workers [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb203e53700 (most recent call first): [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 379 in _recv [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 414 in _recv_bytes [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 250 in recv [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 576 in _handle_results [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.465Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.465Z] [2020-07-02T20:01:21.465Z] Thread 0x00007fb204654700 (most recent call first): [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 528 in _handle_tasks [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.466Z] [2020-07-02T20:01:21.466Z] Thread 0x00007fb204e55700 (most recent call first): [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/selectors.py", line 415 in select [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/connection.py", line 930 in wait [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 499 in _wait_for_updates [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 519 in _handle_workers [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/threading.py", line 870 in run [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap [2020-07-02T20:01:21.466Z] [2020-07-02T20:01:21.466Z] Thread 0x00007fb20b6da740 (most recent call first): [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/linecache.py", line 137 in updatecache [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/linecache.py", line 47 in getlines [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/linecache.py", line 16 in getline [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/traceback.py", line 289 in line [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/traceback.py", line 367 in extract [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/traceback.py", line 213 in extract_stack [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/traceback.py", line 197 in format_stack [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pytest_timeout.py", line 381 in dump_stacks [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pytest_timeout.py", line 320 in timeout_sigalrm [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pytest_timeout.py", line 149 in handler [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 674 in _help_stuff_finish [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/pool.py", line 689 in _terminate_pool [2020-07-02T20:01:21.466Z] File "/usr/lib/python3.8/multiprocessing/util.py", line 201 in __call__ [2020-07-02T20:01:21.466Z] File "/work/mxnet/tests/python/conftest.py", line 43 in check_leak_ndarray [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 784 in call_fixture_func [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 958 in pytest_fixture_setup [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/callers.py", line 187 in _multicall [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 84 in <lambda> [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 93 in _hookexec [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/hooks.py", line 286 in __call__ [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 909 in execute [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 580 in _compute_fixture_value [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 499 in _get_active_fixturedef [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 483 in getfixturevalue [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 473 in _fillfixtures [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/fixtures.py", line 294 in fillfixtures [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1449 in setup [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 370 in prepare [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 122 in pytest_runtest_setup [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/callers.py", line 187 in _multicall [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 84 in <lambda> [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 93 in _hookexec [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/hooks.py", line 286 in __call__ [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 210 in <lambda> [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 237 in from_call [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 209 in call_runtest_hook [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/flaky/flaky_pytest_plugin.py", line 129 in call_and_report [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 93 in runtestprotocol [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/flaky/flaky_pytest_plugin.py", line 92 in pytest_runtest_protocol [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/callers.py", line 187 in _multicall [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 84 in <lambda> [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 93 in _hookexec [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/hooks.py", line 286 in __call__ [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 271 in pytest_runtestloop [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/callers.py", line 187 in _multicall [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 84 in <lambda> [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 93 in _hookexec [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/hooks.py", line 286 in __call__ [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 247 in _main [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 197 in wrap_session [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/main.py", line 240 in pytest_cmdline_main [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/callers.py", line 187 in _multicall [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 84 in <lambda> [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/manager.py", line 93 in _hookexec [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/pluggy/hooks.py", line 286 in __call__ [2020-07-02T20:01:21.466Z] File "/usr/local/lib/python3.8/dist-packages/_pytest/config/__init__.py", line 92 in main [2020-07-02T20:01:21.466Z] File "/usr/local/bin/pytest", line 11 in <module> [2020-07-02T22:59:11.698Z] Sending interrupt signal to process [2020-07-02T22:59:18.828Z] 2020-07-02 22:59:11,751 - root - WARNING - Signal 15 received, cleaning up... [2020-07-02T22:59:18.828Z] 2020-07-02 22:59:11,752 - root - WARNING - Cleaning up containers [2020-07-02T22:59:18.828Z] 2020-07-02 22:59:15,573 - root - INFO - ☠: stopped container 8db2488e60e1 [2020-07-02T22:59:18.828Z] 2020-07-02 22:59:15,810 - root - INFO - 🚽: removed container 8db2488e60e1 [2020-07-02T22:59:18.828Z] 2020-07-02 22:59:15,810 - root - INFO - Cleaning up containers finished. [2020-07-02T22:59:18.828Z] 2020-07-02 22:59:15,810 - root - WARNING - done. Exiting with error. ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
