Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16800
Change subject: IMPALA-10369: Dump server stacktraces when test_concurrent_ddls.py timeout ...................................................................... IMPALA-10369: Dump server stacktraces when test_concurrent_ddls.py timeout Recently, we see many timeout failures of test_concurrent_ddls.py in S3 builds, e.g. IMPALA-10280, IMPALA-10301, IMPALA-10363. It'd be helpful to dump the server stacktraces so we can understand why some RPCs are slow/stuck. This patch extracts the logic of dumping stacktraces in script-timeout-check.sh to a separate script, script-timeout-check.sh. The script also dumps jstacks of HMS and NameNode. Dumping all these stacktraces is time-consuming so we do them in parallel, which also helps to get consistent snapshots of all servers. When any tests in test_concurrent_ddls.py timeout, we use dump-stacktraces.sh to dump the stacktraces before exit. Previously, some tests depend on pytest.mark.timeout for detecting timeouts. It's hard to add a customized callback for dumping server stacktraces. So this patch refactors test_concurrent_ddls.py to only use timeout of multiprocessing. Tests: - Tested the scripts locally. - (WIP) Run jenkins jobs. Change-Id: I514cf2d0ff842805c0abf7211f2a395151174173 --- A bin/dump-stacktraces.sh M bin/script-timeout-check.sh M tests/custom_cluster/test_concurrent_ddls.py M tests/util/shell_util.py 4 files changed, 103 insertions(+), 38 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/16800/1 -- To view, visit http://gerrit.cloudera.org:8080/16800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I514cf2d0ff842805c0abf7211f2a395151174173 Gerrit-Change-Number: 16800 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
