[
https://issues.apache.org/jira/browse/IMPALA-10039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172663#comment-17172663
]
Wenzhe Zhou commented on IMPALA-10039:
--------------------------------------
Thanks Joe for the script which can easily reproduce the issue.
Recent patch for [IMPALA-5746|http://issues.apache.org/jira/browse/IMPALA-5746]
registers a callback function for the updating of cluster membership. The
callback function cancels the queries scheduled by the failed coordinators.
This callback function was called during Expr-test running. In some cases, it
make QueryState::Cancel() get called before thread unsafe function
QueryState::Init() is completed. Hence QueryState::Cancel() is called with
instances_prepared_barrier_ as nullptr, and cause crash.
There is another dead-lock. If QueryState::Cancel() is called right after
QueryState::Init() return error with not null instances_prepared_barrier_ ,
QueryState::Cancel() wait on instances_prepared_barrier_ forever since
fragment instances are not executed and instances_prepared_barrier_ will not be
notified.
To fix it, we should make QueryState::Cancel() to wait until QueryState::Init()
is completed, and reset instances_prepared_barrier_ if Init() failed. Also
checks if the process running for tests and only registers the callback
function if it's not running for BE/FE tests.
> Expr-test crash in ExprTest.LiteralExprs during core run
> --------------------------------------------------------
>
> Key: IMPALA-10039
> URL: https://issues.apache.org/jira/browse/IMPALA-10039
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Laszlo Gaal
> Assignee: Wenzhe Zhou
> Priority: Blocker
> Labels: broken-build
>
> Expr-test crashed with a minidump during a core-mode run.
> The test log:
> {code}
> 22/123 Test #22: expr-test ........................***Failed 4.42 sec
> Turning perftools heap leak checking off
> seed = 1596358469
> Note: Google Test filter = Instantiations/ExprTest.*
> [==========] Running 192 tests from 1 test case.
> [----------] Global test environment set-up.
> [----------] 192 tests from Instantiations/ExprTest
> 20/08/02 01:54:29 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> Running without optimization passes.
> [ RUN ] Instantiations/ExprTest.NullLiteral/0
> [ OK ] Instantiations/ExprTest.NullLiteral/0 (1 ms)
> [ RUN ] Instantiations/ExprTest.NullLiteral/1
> [ OK ] Instantiations/ExprTest.NullLiteral/1 (1 ms)
> [ RUN ] Instantiations/ExprTest.NullLiteral/2
> [ OK ] Instantiations/ExprTest.NullLiteral/2 (0 ms)
> [ RUN ] Instantiations/ExprTest.LiteralConstruction/0
> [ OK ] Instantiations/ExprTest.LiteralConstruction/0 (4 ms)
> [ RUN ] Instantiations/ExprTest.LiteralConstruction/1
> [ OK ] Instantiations/ExprTest.LiteralConstruction/1 (1 ms)
> [ RUN ] Instantiations/ExprTest.LiteralConstruction/2
> [ OK ] Instantiations/ExprTest.LiteralConstruction/2 (2 ms)
> [ RUN ] Instantiations/ExprTest.LiteralExprs/0
> Wrote minidump to
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/be_tests/minidumps/unifiedbetests/3c669d32-0e5a-42d6-ae70e79b-9f91038f.dmp
> Wrote minidump to
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/be_tests/minidumps/unifiedbetests/3c669d32-0e5a-42d6-ae70e79b-9f91038f.dmp
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007f1e95c21c30, pid=4127, tid=0x00007f1e3d322700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build
> 1.8.0_144-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C [libpthread.so.0+0x9c30] pthread_mutex_lock+0x0
> #
> # Core dump written. Default location:
> /data0/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/exprs/core
> or core.4127
> #
> # An error report file with more information is saved as:
> #
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/hs_err_pid4127.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> #
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/build/debug//exprs/expr-test:
> line 10: 4127 Aborted (core dumped)
> ${IMPALA_HOME}/bin/run-jvm-binary.sh
> ${IMPALA_HOME}/be/build/latest/service/unifiedbetests
> --gtest_filter=${GTEST_FILTER}
> --gtest_output=xml:${IMPALA_BE_TEST_LOGS_DIR}/${TEST_EXEC_NAME}.xml
> -log_filename="${TEST_EXEC_NAME}" "$@"
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-asf-master-core/repos/Impala/bin/junitxml_prune_notrun.py",
> line 71, in <module>
> if __name__ == "__main__": main()
> File
> "/data/jenkins/workspace/impala-asf-master-core/repos/Impala/bin/junitxml_prune_notrun.py",
> line 68, in main
> junitxml_prune_notrun(options.filename)
> File
> "/data/jenkins/workspace/impala-asf-master-core/repos/Impala/bin/junitxml_prune_notrun.py",
> line 31, in junitxml_prune_notrun
> root = tree.parse(junitxml_filename)
> File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 647, in parse
> source = open(source, "rb")
> IOError: [Errno 2] No such file or directory:
> '/data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/be_tests/expr-test.xml'
> {code}
> Minidump backtrace:
> {code}
> #0 0x00007f1e925f31f7 in raise () from /lib64/libc.so.6
> #1 0x00007f1e925f48e8 in abort () from /lib64/libc.so.6
> #2 0x00007f1e9532d185 in os::abort(bool) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #3 0x00007f1e954cf593 in VMError::report_and_die() () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #4 0x00007f1e9533268f in JVM_handle_linux_signal () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #5 0x00007f1e95328be3 in signalHandler(int, siginfo*, void*) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #6 <signal handler called>
> #7 0x00007f1e95c21c30 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #8 0x0000000002217537 in __gthread_mutex_lock (__mutex=0x38) at
> /data0/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/include/c++/7.5.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748
> #9 0x0000000002233f7c in std::mutex::lock (this=0x38) at
> /data0/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/include/c++/7.5.0/bits/std_mutex.h:103
> #10 0x000000000223bb4b in std::unique_lock<std::mutex>::lock
> (this=0x7f1e3d3212e0) at
> /data0/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/include/c++/7.5.0/bits/std_mutex.h:267
> #11 0x0000000002238fae in std::unique_lock<std::mutex>::unique_lock
> (this=0x7f1e3d3212e0, __m=...) at
> /data0/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/include/c++/7.5.0/bits/std_mutex.h:197
> #12 0x000000000314f4b4 in impala::Promise<bool, (impala::PromiseMode)0>::Get
> (this=0x0) at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/promise.h:85
> #13 0x0000000003408022 in impala::TypedCountingBarrier<bool>::Wait (this=0x0)
> at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/counting-barrier.h:61
> #14 0x0000000003407a68 in impala::CountingBarrier::Wait (this=0x0) at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/counting-barrier.h:97
> #15 0x0000000003402ddc in impala::QueryState::WaitForPrepare (this=0xe9d0900)
> at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/runtime/query-state.cc:671
> #16 0x0000000003405028 in impala::QueryState::Cancel (this=0xe9d0900) at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/runtime/query-state.cc:850
> #17 0x00000000033f43cd in impala::QueryExecMgr::CancelFromThreadPool
> (this=0x132b8400, cancellation_task=...) at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/runtime/query-exec-mgr.cc:262
> #18 0x00000000033fb107 in boost::_mfi::mf1<void, impala::QueryExecMgr,
> impala::QueryExecMgr::QueryCancellationTask const&>::operator()
> (this=0x11d3ac58, p=0x132b8400, a1=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:165
> #19 0x00000000033fa56f in
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::arg<2>
> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr,
> impala::QueryExecMgr::QueryCancellationTask const&>, boost::_bi::rrlist2<int,
> impala::QueryExecMgr::QueryCancellationTask const&> > (this=0x11d3ac68,
> f=..., a=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:319
> #20 0x00000000033f98a4 in boost::_bi::bind_t<void, boost::_mfi::mf1<void,
> impala::QueryExecMgr, impala::QueryExecMgr::QueryCancellationTask const&>,
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::arg<2> >
> >::operator()<int, impala::QueryExecMgr::QueryCancellationTask const&>(int&&,
> impala::QueryExecMgr::QueryCancellationTask const&) (this=0x11d3ac58,
> a1=@0x7f1e3d3215f4: 0, a2=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1246
> #21 0x00000000033f892a in
> boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void,
> boost::_mfi::mf1<void, impala::QueryExecMgr,
> impala::QueryExecMgr::QueryCancellationTask const&>,
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::arg<2> >
> >, void, int, impala::QueryExecMgr::QueryCancellationTask const&>::invoke
> (function_obj_ptr=..., a0=0, a1=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
> #22 0x00000000033f79e7 in boost::function2<void, int,
> impala::QueryExecMgr::QueryCancellationTask const&>::operator()
> (this=0x11d3ac50, a0=0, a1=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770
> #23 0x00000000033f61a1 in
> impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>::WorkerThread
> (this=0x11d3ac00, thread_id=0) at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/thread-pool.h:166
> #24 0x00000000033fbdb8 in boost::_mfi::mf1<void,
> impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>,
> int>::operator() (this=0x129de000, p=0x11d3ac00, a1=0) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:165
> #25 0x00000000033fb9f8 in
> boost::_bi::list2<boost::_bi::value<impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>*>,
> boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void,
> impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>, int>,
> boost::_bi::list0> (this=0x129de010, f=..., a=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:319
> #26 0x00000000033fb1cd in boost::_bi::bind_t<void, boost::_mfi::mf1<void,
> impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>, int>,
> boost::_bi::list2<boost::_bi::value<impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>*>,
> boost::_bi::value<int> > >::operator() (this=0x129de000) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222
> #27 0x00000000033fa961 in
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf1<void,
> impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>, int>,
> boost::_bi::list2<boost::_bi::value<impala::ThreadPool<impala::QueryExecMgr::QueryCancellationTask>*>,
> boost::_bi::value<int> > >, void>::invoke (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
> #28 0x0000000003237382 in boost::function0<void>::operator()
> (this=0x7f1e3d321ba0) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770
> #29 0x000000000446aac6 in
> impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=...,
> functor=..., parent_thread_info=0x0, thread_started=0x7fffae137320) at
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/thread.cc:360
> #30 0x00000000044731c9 in
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0x13d4c380,
> f=@0x13d4c378: 0x446a780
> <impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:531
> #31 0x00000000044730ed in boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()() (this=0x13d4c378) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222
> #32 0x00000000044730ae in boost::detail::thread_data<boost::_bi::bind_t<void,
> void (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run() (this=0x13d4c1c0) at
> /data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116
> #33 0x0000000004714232 in thread_proxy ()
> #34 0x00007f1e95c1fe25 in start_thread () from /lib64/libpthread.so.0
> #35 0x00007f1e926b634d in clone () from /lib64/libc.so.6
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
