[
https://issues.apache.org/jira/browse/IMPALA-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811064#comment-17811064
]
Michael Smith commented on IMPALA-12757:
----------------------------------------
https://github.com/google/sanitizers/issues/814 might also be related. I
haven't worked through the circumstances enough to tell yet.
> TSAN flags lock-order-inversion during internal-server-test
> -----------------------------------------------------------
>
> Key: IMPALA-12757
> URL: https://issues.apache.org/jira/browse/IMPALA-12757
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.4.0
> Reporter: Michael Smith
> Priority: Major
>
> internal-server-test has a tight loop starting queries and fetching /queries.
> That's led to identifying several latent threading issues. Once IMPALA-12747
> is addressed, TSAN identifies a new error:
> {code}
> $ run-jvm-binary.sh ./be/build/debug/service/internal-server-test
> I20240125 10:44:44.971467 152733 openssl_util.cc:110] FIPS mode is disabled.
> Picked up JAVA_TOOL_OPTIONS: -Dsun.java.command=internal-server-test
> 24/01/25 10:44:45 WARN fs.FileSystem: Cannot load filesystem:
> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
> Provider org.apache.hadoop.hive.ql.io.NullScanFileSystem not found
> 24/01/25 10:44:45 WARN fs.FileSystem: Cannot load filesystem:
> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
> Provider org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem not found
> 24/01/25 10:44:45 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [==========] Running 10 tests from 1 test case.
> [----------] Global test environment set-up.
> [----------] 10 tests from InternalServerTest
> [ RUN ] InternalServerTest.QueryTimeout
> [ OK ] InternalServerTest.QueryTimeout (10236 ms)
> [ RUN ] InternalServerTest.InvalidQueryOption
> [ OK ] InternalServerTest.InvalidQueryOption (76 ms)
> [ RUN ] InternalServerTest.MultipleQueriesMultipleSessions
> /home/michael/Impala/be/src/service/internal-server-test.cc:289: Failure
> Value of: status_.ok()
> Actual: false
> Expected: true
> Error: Failed due to unreachable impalad(s): michaelsmith-22742:27000
> [ FAILED ] InternalServerTest.MultipleQueriesMultipleSessions (17225 ms)
> [ RUN ] InternalServerTest.RetryFailedQuery
> [ OK ] InternalServerTest.RetryFailedQuery (1206 ms)
> [ RUN ] InternalServerTest.MultipleQueriesOneSession
> ^C==================
> WARNING: ThreadSanitizer: thread leak (pid=152733)
> Thread T1086 (tid=154103, finished) created by main thread at:
> #0 pthread_create <null> (internal-server-test+0x203c383)
> #1 boost::thread::start_thread_noexcept() <null>
> (internal-server-test+0x3bcd3fd)
> #2 boost::thread::thread<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> >, std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*,
> impala::Promise<long, (impala::PromiseMode)0>*>(void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> >, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long,
> (impala::PromiseMode)0>*)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:424:13
> (internal-server-test+0x2d84514)
> #3 impala::Thread::StartThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread,
> std::default_delete<impala::Thread> >*, bool)
> /home/michael/Impala/be/src/util/thread.cc:317:13
> (internal-server-test+0x2d8091c)
> #4 impala::Status impala::Thread::Create<void
> (impala::ClientRequestState::*)(),
> impala::ClientRequestState*>(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, void (impala::ClientRequestState::* const&)(),
> impala::ClientRequestState* const&, std::unique_ptr<impala::Thread,
> std::default_delete<impala::Thread> >*, bool)
> /home/michael/Impala/be/src/util/thread.h:81:12
> (internal-server-test+0x2b382f7)
> #5 impala::ClientRequestState::WaitAsync()
> /home/michael/Impala/be/src/service/client-request-state.cc:1126:10
> (internal-server-test+0x2b2f9f4)
> #6 impala::ImpalaServer::WaitForResults(impala::TUniqueId&)
> /home/michael/Impala/be/src/service/internal-server.cc:156:3
> (internal-server-test+0x2ad148e)
> #7 non-virtual thunk to
> impala::ImpalaServer::WaitForResults(impala::TUniqueId&)
> /home/michael/Impala/be/src/service/internal-server.cc
> (internal-server-test+0x2ad15fd)
> #8
> impala::internalservertest::InternalServerTest_MultipleQueriesOneSession_Test::TestBody()
> /home/michael/Impala/be/src/service/internal-server-test.cc:380:3
> (internal-server-test+0x209fb9c)
> #9 void
> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
> void>(testing::Test*, void (testing::Test::*)(), char const*) <null>
> (internal-server-test+0x5212c1c)
> #10 <null> <null> (libc.so.6+0x29d8f)
> SUMMARY: ThreadSanitizer: thread leak
> (/home/michael/Impala/be/build/debug/service/internal-server-test+0x203c383)
> in __interceptor_pthread_create
> ==================
> michael@michaelsmith-22742 ~/Impala ❯❯❯ TSAN_OPTIONS=second_deadlock_stack=1
> run-jvm-binary.sh ./be/build/debug/service/internal-server-test
> michael@michaelsmith-22742 ~/Impala ❯❯❯ export
> TSAN_OPTIONS=second_deadlock_stack=1 ✘ 130
> michael@michaelsmith-22742 ~/Impala ❯❯❯ run-jvm-binary.sh
> ./be/build/debug/service/internal-server-test
> I20240125 10:46:49.913501 161734 openssl_util.cc:110] FIPS mode is disabled.
> Picked up JAVA_TOOL_OPTIONS: -Dsun.java.command=internal-server-test
> 24/01/25 10:46:50 WARN fs.FileSystem: Cannot load filesystem:
> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
> Provider org.apache.hadoop.hive.ql.io.NullScanFileSystem not found
> 24/01/25 10:46:50 WARN fs.FileSystem: Cannot load filesystem:
> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
> Provider org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem not found
> 24/01/25 10:46:50 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [==========] Running 10 tests from 1 test case.
> [----------] Global test environment set-up.
> [----------] 10 tests from InternalServerTest
> [ RUN ] InternalServerTest.QueryTimeout
> [ OK ] InternalServerTest.QueryTimeout (10149 ms)
> [ RUN ] InternalServerTest.InvalidQueryOption
> [ OK ] InternalServerTest.InvalidQueryOption (75 ms)
> [ RUN ] InternalServerTest.MultipleQueriesMultipleSessions
> /home/michael/Impala/be/src/service/internal-server-test.cc:289: Failure
> Value of: status_.ok()
> Actual: false
> Expected: true
> Error: Failed due to unreachable impalad(s): michaelsmith-22742:27000
> [ FAILED ] InternalServerTest.MultipleQueriesMultipleSessions (17058 ms)
> [ RUN ] InternalServerTest.RetryFailedQuery
> [ OK ] InternalServerTest.RetryFailedQuery (1205 ms)
> [ RUN ] InternalServerTest.MultipleQueriesOneSession
> /home/michael/Impala/be/src/service/internal-server-test.cc:381: Failure
> Value of: status_.ok()
> Actual: false
> Expected: true
> Error: Failed due to unreachable impalad(s): michaelsmith-22742:27001
> [ FAILED ] InternalServerTest.MultipleQueriesOneSession (15934 ms)
> [ RUN ] InternalServerTest.MissingClosingQuote
> [ OK ] InternalServerTest.MissingClosingQuote (8436 ms)
> [ RUN ] InternalServerTest.SyntaxError
> [ OK ] InternalServerTest.SyntaxError (7929 ms)
> [ RUN ] InternalServerTest.UnclosedComment
> [ OK ] InternalServerTest.UnclosedComment (96 ms)
> [ RUN ] InternalServerTest.TableNotExist
> [ OK ] InternalServerTest.TableNotExist (7994 ms)
> [ RUN ] InternalServerTest.SimultaneousMultipleQueriesOneSession
> ==================
> WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock)
> (pid=161734)
> Cycle in lock order graph: M49465 (0x7b50005302b4) => M49694
> (0x7b100017f5d0) => M49465
> Mutex M49694 acquired here while holding mutex M49465 in thread T1310:
> #0 pthread_mutex_lock <null> (internal-server-test+0x202dbb0)
> #1 __gthread_mutex_lock(pthread_mutex_t*)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749:12
> (internal-server-test+0x20a41b6)
> #2 std::mutex::lock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/std_mutex.h:100:17
> (internal-server-test+0x20ac089)
> #3 std::unique_lock<std::mutex>::lock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17
> (internal-server-test+0x25e1c9f)
> #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:68:2
> (internal-server-test+0x25e1013)
> #5 std::cv_status
> std::_V2::condition_variable_any::wait_until<std::unique_lock<impala::SpinLock>,
> std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l,
> 1000000000l> > >(std::unique_lock<impala::SpinLock>&,
> std::chrono::time_point<std::chrono::_V2::steady_clock,
> std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:342:21
> (internal-server-test+0x2731a45)
> #6 std::cv_status
> std::_V2::condition_variable_any::wait_for<std::unique_lock<impala::SpinLock>,
> long, std::ratio<1l, 1000l> >(std::unique_lock<impala::SpinLock>&,
> std::chrono::duration<long, std::ratio<1l, 1000l> > const&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:366:16
> (internal-server-test+0x272b816)
> #7
> impala::KrpcDataStreamSender::Channel::WaitForRpcLocked(std::unique_lock<impala::SpinLock>*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:437:18
> (internal-server-test+0x2721637)
> #8 impala::KrpcDataStreamSender::Channel::WaitForRpc()
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:425:10
> (internal-server-test+0x27214e8)
> #9 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5
> (internal-server-test+0x272a1ca)
> #10 impala::FragmentInstanceState::ExecInternal()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3
> (internal-server-test+0x2711440)
> #11 impala::FragmentInstanceState::Exec()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14
> (internal-server-test+0x270f29c)
> #12 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /home/michael/Impala/be/src/runtime/query-state.cc:991:24
> (internal-server-test+0x2646f48)
> #13 impala::QueryState::StartFInstances()::$_15::operator()() const
> /home/michael/Impala/be/src/runtime/query-state.cc:896:37
> (internal-server-test+0x264bae2)
> #14
> boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15,
> void>::invoke(boost::detail::function::function_buffer&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11
> (internal-server-test+0x264b899)
> #15 boost::function0<void>::operator()() const
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (internal-server-test+0x2592ff1)
> #16 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /home/michael/Impala/be/src/util/thread.cc:360:3
> (internal-server-test+0x2d80f98)
> #17 void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9
> (internal-server-test+0x2d89d1c)
> #18 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (internal-server-test+0x2d89c33)
> #19 boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17
> (internal-server-test+0x2d89920)
> #20 thread_proxy <null> (internal-server-test+0x3bce0f6)
> Mutex M49465 previously acquired by the same thread here:
> #0 AnnotateRWLockAcquired <null> (internal-server-test+0x205502c)
> #1 base::SpinLock::Lock()
> /home/michael/Impala/be/src/gutil/spinlock.h:77:5
> (internal-server-test+0x215d515)
> #2 impala::SpinLock::lock()
> /home/michael/Impala/be/src/util/spinlock.h:34:8
> (internal-server-test+0x215d4a9)
> #3 std::unique_lock<impala::SpinLock>::lock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17
> (internal-server-test+0x2610cff)
> #4 std::unique_lock<impala::SpinLock>::unique_lock(impala::SpinLock&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:68:2
> (internal-server-test+0x2610663)
> #5 impala::KrpcDataStreamSender::Channel::WaitForRpc()
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:424:30
> (internal-server-test+0x27214d9)
> #6 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5
> (internal-server-test+0x272a1ca)
> #7 impala::FragmentInstanceState::ExecInternal()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3
> (internal-server-test+0x2711440)
> #8 impala::FragmentInstanceState::Exec()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14
> (internal-server-test+0x270f29c)
> #9 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /home/michael/Impala/be/src/runtime/query-state.cc:991:24
> (internal-server-test+0x2646f48)
> #10 impala::QueryState::StartFInstances()::$_15::operator()() const
> /home/michael/Impala/be/src/runtime/query-state.cc:896:37
> (internal-server-test+0x264bae2)
> #11
> boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15,
> void>::invoke(boost::detail::function::function_buffer&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11
> (internal-server-test+0x264b899)
> #12 boost::function0<void>::operator()() const
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (internal-server-test+0x2592ff1)
> #13 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /home/michael/Impala/be/src/util/thread.cc:360:3
> (internal-server-test+0x2d80f98)
> #14 void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9
> (internal-server-test+0x2d89d1c)
> #15 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (internal-server-test+0x2d89c33)
> #16 boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17
> (internal-server-test+0x2d89920)
> #17 thread_proxy <null> (internal-server-test+0x3bce0f6)
> Mutex M49465 acquired here while holding mutex M49694 in thread T1310:
> #0 AnnotateRWLockAcquired <null> (internal-server-test+0x205502c)
> #1 base::SpinLock::Lock()
> /home/michael/Impala/be/src/gutil/spinlock.h:77:5
> (internal-server-test+0x215d515)
> #2 impala::SpinLock::lock()
> /home/michael/Impala/be/src/util/spinlock.h:34:8
> (internal-server-test+0x215d4a9)
> #3 std::unique_lock<impala::SpinLock>::lock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17
> (internal-server-test+0x2610cff)
> #4
> std::_V2::condition_variable_any::_Unlock<std::unique_lock<impala::SpinLock>
> >::~_Unlock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:283:14
> (internal-server-test+0x2613c16)
> #5 std::cv_status
> std::_V2::condition_variable_any::wait_until<std::unique_lock<impala::SpinLock>,
> std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l,
> 1000000000l> > >(std::unique_lock<impala::SpinLock>&,
> std::chrono::time_point<std::chrono::_V2::steady_clock,
> std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:348:7
> (internal-server-test+0x2731a8b)
> #6 std::cv_status
> std::_V2::condition_variable_any::wait_for<std::unique_lock<impala::SpinLock>,
> long, std::ratio<1l, 1000l> >(std::unique_lock<impala::SpinLock>&,
> std::chrono::duration<long, std::ratio<1l, 1000l> > const&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:366:16
> (internal-server-test+0x272b816)
> #7
> impala::KrpcDataStreamSender::Channel::WaitForRpcLocked(std::unique_lock<impala::SpinLock>*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:437:18
> (internal-server-test+0x2721637)
> #8 impala::KrpcDataStreamSender::Channel::WaitForRpc()
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:425:10
> (internal-server-test+0x27214e8)
> #9 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5
> (internal-server-test+0x272a1ca)
> #10 impala::FragmentInstanceState::ExecInternal()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3
> (internal-server-test+0x2711440)
> #11 impala::FragmentInstanceState::Exec()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14
> (internal-server-test+0x270f29c)
> #12 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /home/michael/Impala/be/src/runtime/query-state.cc:991:24
> (internal-server-test+0x2646f48)
> #13 impala::QueryState::StartFInstances()::$_15::operator()() const
> /home/michael/Impala/be/src/runtime/query-state.cc:896:37
> (internal-server-test+0x264bae2)
> #14
> boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15,
> void>::invoke(boost::detail::function::function_buffer&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11
> (internal-server-test+0x264b899)
> #15 boost::function0<void>::operator()() const
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (internal-server-test+0x2592ff1)
> #16 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /home/michael/Impala/be/src/util/thread.cc:360:3
> (internal-server-test+0x2d80f98)
> #17 void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9
> (internal-server-test+0x2d89d1c)
> #18 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (internal-server-test+0x2d89c33)
> #19 boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17
> (internal-server-test+0x2d89920)
> #20 thread_proxy <null> (internal-server-test+0x3bce0f6)
> Mutex M49694 previously acquired by the same thread here:
> #0 pthread_mutex_lock <null> (internal-server-test+0x202dbb0)
> #1 __gthread_mutex_lock(pthread_mutex_t*)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749:12
> (internal-server-test+0x20a41b6)
> #2 std::mutex::lock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/std_mutex.h:100:17
> (internal-server-test+0x20ac089)
> #3 std::unique_lock<std::mutex>::lock()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:138:17
> (internal-server-test+0x25e1c9f)
> #4 std::unique_lock<std::mutex>::unique_lock(std::mutex&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_lock.h:68:2
> (internal-server-test+0x25e1013)
> #5 std::cv_status
> std::_V2::condition_variable_any::wait_until<std::unique_lock<impala::SpinLock>,
> std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l,
> 1000000000l> > >(std::unique_lock<impala::SpinLock>&,
> std::chrono::time_point<std::chrono::_V2::steady_clock,
> std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:342:21
> (internal-server-test+0x2731a45)
> #6 std::cv_status
> std::_V2::condition_variable_any::wait_for<std::unique_lock<impala::SpinLock>,
> long, std::ratio<1l, 1000l> >(std::unique_lock<impala::SpinLock>&,
> std::chrono::duration<long, std::ratio<1l, 1000l> > const&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/condition_variable:366:16
> (internal-server-test+0x272b816)
> #7
> impala::KrpcDataStreamSender::Channel::WaitForRpcLocked(std::unique_lock<impala::SpinLock>*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:437:18
> (internal-server-test+0x2721637)
> #8 impala::KrpcDataStreamSender::Channel::WaitForRpc()
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:425:10
> (internal-server-test+0x27214e8)
> #9 impala::KrpcDataStreamSender::FlushFinal(impala::RuntimeState*)
> /home/michael/Impala/be/src/runtime/krpc-data-stream-sender.cc:1188:5
> (internal-server-test+0x272a1ca)
> #10 impala::FragmentInstanceState::ExecInternal()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:466:3
> (internal-server-test+0x2711440)
> #11 impala::FragmentInstanceState::Exec()
> /home/michael/Impala/be/src/runtime/fragment-instance-state.cc:104:14
> (internal-server-test+0x270f29c)
> #12 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /home/michael/Impala/be/src/runtime/query-state.cc:991:24
> (internal-server-test+0x2646f48)
> #13 impala::QueryState::StartFInstances()::$_15::operator()() const
> /home/michael/Impala/be/src/runtime/query-state.cc:896:37
> (internal-server-test+0x264bae2)
> #14
> boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_15,
> void>::invoke(boost::detail::function::function_buffer&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11
> (internal-server-test+0x264b899)
> #15 boost::function0<void>::operator()() const
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (internal-server-test+0x2592ff1)
> #16 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /home/michael/Impala/be/src/util/thread.cc:360:3
> (internal-server-test+0x2d80f98)
> #17 void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9
> (internal-server-test+0x2d89d1c)
> #18 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (internal-server-test+0x2d89c33)
> #19 boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17
> (internal-server-test+0x2d89920)
> #20 thread_proxy <null> (internal-server-test+0x3bce0f6)
> Thread T1137 (tid=163645, running) created by thread T1134 at:
> #0 pthread_create <null> (internal-server-test+0x203c383)
> #1 boost::thread::start_thread_noexcept() <null>
> (internal-server-test+0x3bcd3fd)
> #2 boost::thread::thread<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> >, std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*,
> impala::Promise<long, (impala::PromiseMode)0>*>(void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> >, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long,
> (impala::PromiseMode)0>*)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:424:13
> (internal-server-test+0x2d84514)
> #3 impala::Thread::StartThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread,
> std::default_delete<impala::Thread> >*, bool)
> /home/michael/Impala/be/src/util/thread.cc:317:13
> (internal-server-test+0x2d8091c)
> #4 impala::Status
> impala::Thread::Create<impala::QueryState::StartFInstances()::$_15>(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, impala::QueryState::StartFInstances()::$_15 const&,
> std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool)
> /home/michael/Impala/be/src/util/thread.h:74:12
> (internal-server-test+0x26468aa)
> #5 impala::QueryState::StartFInstances()
> /home/michael/Impala/be/src/runtime/query-state.cc:895:11
> (internal-server-test+0x264623e)
> #6 impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*)
> /home/michael/Impala/be/src/runtime/query-exec-mgr.cc:162:7
> (internal-server-test+0x26347d9)
> #7 boost::_mfi::mf1<void, impala::QueryExecMgr,
> impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*)
> const
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/mem_fn_template.hpp:165:29
> (internal-server-test+0x263c0bd)
> #8 void boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>,
> boost::_bi::value<impala::QueryState*> >::operator()<boost::_mfi::mf1<void,
> impala::QueryExecMgr, impala::QueryState*>,
> boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void,
> impala::QueryExecMgr, impala::QueryState*>&, boost::_bi::list0&, int)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:319:9
> (internal-server-test+0x263c00a)
> #9 boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr,
> impala::QueryState*>,
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>,
> boost::_bi::value<impala::QueryState*> > >::operator()()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (internal-server-test+0x263bf73)
> #10
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>,
> boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>,
> boost::_bi::value<impala::QueryState*> > >,
> void>::invoke(boost::detail::function::function_buffer&)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:158:11
> (internal-server-test+0x263bcb1)
> #11 boost::function0<void>::operator()() const
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
> (internal-server-test+0x2592ff1)
> #12 impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /home/michael/Impala/be/src/util/thread.cc:360:3
> (internal-server-test+0x2d80f98)
> #13 void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:531:9
> (internal-server-test+0x2d89d1c)
> #14 boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/bind/bind.hpp:1294:16
> (internal-server-test+0x2d89c33)
> #15 boost::detail::thread_data<boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run()
> /home/michael/Impala/toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:120:17
> (internal-server-test+0x2d89920)
> #16 thread_proxy <null> (internal-server-test+0x3bce0f6)
> SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock)
> (/home/michael/Impala/be/build/debug/service/internal-server-test+0x202dbb0)
> in __interceptor_pthread_mutex_lock
> ==================
> {code}
> That looks similar to
> https://stackoverflow.com/questions/71188546/threadsanitizer-double-lock-of-a-mutex-with-stdjthread,
> and I'm not sure I understand it yet. I want to address it as a separate
> effort.
> I encountered this on Ubuntu 22.04 (x86_64). Didn't see it reproduce on
> CentOS 7.9.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
