[
https://issues.apache.org/jira/browse/IMPALA-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-9879:
-------------------------------------
Assignee: Tim Armstrong
> ASAN use-after-free with KRPC thread and
> Coordinator::FilterState::ApplyUpdate()
> ---------------------------------------------------------------------------------
>
> Key: IMPALA-9879
> URL: https://issues.apache.org/jira/browse/IMPALA-9879
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0
> Reporter: Joe McDonnell
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build
>
> An ASAN core run failed with the following Impalad crash:
>
> {noformat}
> ==4348==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x7fc144423800 at pc 0x000001a50071 bp 0x7fc26d7daa40 sp 0x7fc26d7da1f0
> READ of size 1048576 at 0x7fc144423800 thread T81 (rpc reactor-464)
> #0 0x1a50070 in read_iovec(void*, __sanitizer::__sanitizer_iovec*,
> unsigned long, unsigned long)
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
> #1 0x1a666d1 in read_msghdr(void*, __sanitizer::__sanitizer_msghdr*,
> long)
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2781
> #2 0x1a68fb3 in __interceptor_sendmsg
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2796
> #3 0x38074dc in kudu::Socket::Writev(iovec const*, int, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/net/socket.cc:447:3
> #4 0x3411fa5 in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/transfer.cc:227:26
> #5 0x341aa60 in kudu::rpc::Connection::WriteHandler(ev::io&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/connection.cc:802:31
> #6 0x55ef342 in ev_invoke_pending
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x55ef342)
> #7 0x33a4d8c in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:196:3
> #8 0x55f29ef in ev_run
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x55f29ef)
> #9 0x33a4f81 in kudu::rpc::ReactorThread::RunThread()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:497:9
> #10 0x33b66bb in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
> kudu::rpc::ReactorThread>,
> boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >
> >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
> #11 0x21ba196 in boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
> #12 0x21b6089 in kudu::Thread::SuperviseThread(void*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:675:3
> #13 0x7fcabb86be24 in start_thread (/lib64/libpthread.so.0+0x7e24)
> #14 0x7fcab833f34c in __clone (/lib64/libc.so.6+0xf834c)
> 0x7fc144423800 is located 0 bytes inside of 1048577-byte region
> [0x7fc144423800,0x7fc144523801)
> freed by thread T108 here:
> #0 0x1ad6050 in operator delete(void*)
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
> #1 0x7fcab8c425a9 in __gnu_cxx::new_allocator<char>::deallocate(char*,
> unsigned long)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:125
> #2 0x7fcab8c425a9 in std::allocator_traits<std::allocator<char>
> >::deallocate(std::allocator<char>&, char*, unsigned long)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/alloc_traits.h:462
> #3 0x7fcab8c425a9 in std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:226
> #4 0x7fcab8c425a9 in std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:302
> previously allocated by thread T116 here:
> #0 0x1ad52e0 in operator new(unsigned long)
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
> #1 0x1ad9fce in void std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::_M_construct<char
> const*>(char const*, char const*, std::forward_iterator_tag)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/basic_string.tcc:219:14
> #2 0x7fcab8c44994 in void std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char
> const*>(char const*, char const*, std::__false_type)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:236
> #3 0x7fcab8c44994 in void std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::_M_construct<char
> const*>(char const*, char const*)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:255
> #4 0x7fcab8c44994 in std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::basic_string(char const*,
> unsigned long, std::allocator<char> const&)
> /mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:502
> #5 0x34870c5 in
> impala::Coordinator::FilterState::ApplyUpdate(impala::UpdateFilterParamsPB
> const&, impala::Coordinator*, kudu::rpc::RpcContext*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1422:51
> #6 0x3485fe1 in
> impala::Coordinator::UpdateFilter(impala::UpdateFilterParamsPB const&,
> kudu::rpc::RpcContext*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1320:12
> #7 0x28454e5 in
> impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&,
> kudu::rpc::RpcContext*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1462:11
> #8 0x2797955 in
> impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*,
> impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2710:19
> #9 0x272ced5 in
> impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*,
> impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
> #10 0x34089f3 in std::function<void (google::protobuf::Message const*,
> google::protobuf::Message*,
> kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*,
> google::protobuf::Message*, kudu::rpc::RpcContext*) const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706:14
> #11 0x3407ea1 in
> kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
> #12 0x2364cce in impala::ImpalaServicePool::RunThread()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
> #13 0x236d6cb in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
> impala::ImpalaServicePool>,
> boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> >
> >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
> #14 0x21ba196 in boost::function0<void>::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
> #15 0x2b603b9 in
> impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
> #16 0x2b6b7f8 in void
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void
> (*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
> #17 0x2b6b64b in boost::_bi::bind_t<void, void
> (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&, boost::function<void
> ()>, impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
> #18 0x42a7751 in thread_proxy
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x42a7751)
> Thread T81 (rpc reactor-464) created by T0 here:
> #0 0x19faa00 in __interceptor_pthread_create
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x21b5212 in
> kudu::Thread::StartThread(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, boost::function<void ()> const&, unsigned long,
> scoped_refptr<kudu::Thread>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:619:15
> #2 0x33aeba5 in kudu::Status kudu::Thread::Create<void
> (kudu::rpc::ReactorThread::*)(),
> kudu::rpc::ReactorThread*>(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, void (kudu::rpc::ReactorThread::* const&)(),
> kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.h:164:12
> #3 0x33a4838 in kudu::rpc::ReactorThread::Init()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:188:10
> #4 0x33aca72 in kudu::rpc::Reactor::Init()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:762:18
> #5 0x33921bb in kudu::rpc::Messenger::Init()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:447:5
> #6 0x339186e in
> kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:203:3
> #7 0x234a351 in impala::RpcMgr::Init(impala::TNetworkAddress const&)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:151:3
> #8 0x23b4529 in impala::ExecEnv::Init()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/exec-env.cc:385:3
> #9 0x27692b0 in ImpaladMain(int, char**)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impalad-main.cc:73:3
> #10 0x1ad97a8 in main
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
> #11 0x7fcab8268c04 in __libc_start_main
> (/lib64/libc.so.6+0x21c04){noformat}
> The code that is listed for the allocation is this:
>
>
> {noformat}
> kudu::Slice sidecar_slice;
> kudu::Status status = context->GetInboundSidecar(
> params.bloom_filter().directory_sidecar_idx(), &sidecar_slice);
> if (!status.ok()) {
> ...
> } else if (bloom_filter_.always_false()) {
> int64_t heap_space = sidecar_slice.size();
> if (!coord->filter_mem_tracker_->TryConsume(heap_space)) {
> ...
> } else {
> bloom_filter_ = params.bloom_filter();
> bloom_filter_directory_ = sidecar_slice.ToString(); <-------
> }{noformat}
> That assignment should make a copy, based on the Slice::ToString() code. It
> needs to make a copy, because the Slice is pointing into a KRPC buffer. I
> don't think we saw this prior to GCC7, so one theory is that maybe GCC7 is
> doing something that ASAN doesn't quite understand.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
