Quanlong Huang created IMPALA-10260:
---------------------------------------
Summary: heap-use-after-free AddressSanitizer error in aggregating
runtime filters
Key: IMPALA-10260
URL: https://issues.apache.org/jira/browse/IMPALA-10260
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Fang-Yu Rao
Saw the following ASAN failure in an internal build:
{code:java}
==7121==ERROR: AddressSanitizer: heap-use-after-free on address 0x7fec0d74d800
at pc 0x000001ae9f71 bp 0x7fecfe5d7180 sp 0x7fecfe5d6930
READ of size 1048576 at 0x7fec0d74d800 thread T82 (rpc reactor-757)
#0 0x1ae9f70 in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned
long, unsigned long)
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
#1 0x1b005d1 in read_msghdr(void*, __sanitizer::__sanitizer_msghdr*, long)
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2781
#2 0x1b02eb3 in __interceptor_sendmsg
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2796
#3 0x399f54c in kudu::Socket::Writev(iovec const*, int, long*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/net/socket.cc:447:3
#4 0x35afe75 in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/transfer.cc:227:26
#5 0x35b8930 in kudu::rpc::Connection::WriteHandler(ev::io&, int)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/connection.cc:802:31
#6 0x580bd12 in ev_invoke_pending
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x580bd12)
#7 0x3542c9c in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:196:3
#8 0x580f3bf in ev_run
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x580f3bf)
#9 0x3542e91 in kudu::rpc::ReactorThread::RunThread()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:497:9
#10 0x35545cb in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
kudu::rpc::ReactorThread>,
boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >
>::operator()()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
#11 0x23417c6 in boost::function0<void>::operator()() const
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
#12 0x233e039 in kudu::Thread::SuperviseThread(void*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/thread.cc:675:3
#13 0x7ff54bd29e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
#14 0x7ff5487c034c in __clone (/lib64/libc.so.6+0xf834c)
0x7fec0d74d800 is located 0 bytes inside of 1048577-byte region
[0x7fec0d74d800,0x7fec0d84d801)
freed by thread T112 here:
#0 0x1b6ff50 in operator delete(void*)
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
#1 0x7ff5490c35a9 in __gnu_cxx::new_allocator<char>::deallocate(char*,
unsigned long)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:125
#2 0x7ff5490c35a9 in std::allocator_traits<std::allocator<char>
>::deallocate(std::allocator<char>&, char*, unsigned long)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/alloc_traits.h:462
#3 0x7ff5490c35a9 in std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:226
#4 0x7ff5490c35a9 in std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:302
previously allocated by thread T112 here:
#0 0x1b6f1e0 in operator new(unsigned long)
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
#1 0x1b73ece in void std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char
const*, char const*, std::forward_iterator_tag)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/basic_string.tcc:219:14
#2 0x7ff5490c5994 in void std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char
const*>(char const*, char const*, std::__false_type)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:236
#3 0x7ff5490c5994 in void std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char
const*, char const*)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:255
#4 0x7ff5490c5994 in std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::basic_string(char const*,
unsigned long, std::allocator<char> const&)
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:502
#5 0x3625e65 in
impala::Coordinator::FilterState::ApplyUpdate(impala::UpdateFilterParamsPB
const&, impala::Coordinator*, kudu::rpc::RpcContext*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1469:51
#6 0x3624d81 in
impala::Coordinator::UpdateFilter(impala::UpdateFilterParamsPB const&,
kudu::rpc::RpcContext*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1367:12
#7 0x2964a65 in
impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&,
kudu::rpc::RpcContext*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/client-request-state.cc:1476:11
#8 0x28b5565 in
impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*,
impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impala-server.cc:2855:19
#9 0x2845505 in
impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*,
impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
#10 0x35a68c3 in std::function<void (google::protobuf::Message const*,
google::protobuf::Message*,
kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*,
google::protobuf::Message*, kudu::rpc::RpcContext*) const
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706:14
#11 0x35a5d71 in
kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
#12 0x2435dde in impala::ImpalaServicePool::RunThread()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:280:15
#13 0x243eb8b in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
impala::ImpalaServicePool>,
boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> >
>::operator()()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
#14 0x23417c6 in boost::function0<void>::operator()() const
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
#15 0x2ca64e9 in
impala::Thread::SuperviseThread(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.cc:360:3
#16 0x2cb1928 in void
boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > >,
boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
>::operator()<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, boost::function<void
()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void
(*&)(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, boost::function<void
()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), boost::_bi::list0&, int)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
#17 0x2cb177b in boost::_bi::bind_t<void, void
(*)(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, boost::function<void
()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*),
boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > >,
boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >,
boost::_bi::value<impala::ThreadDebugInfo*>,
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
>::operator()()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
#18 0x44c41f1 in thread_proxy
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x44c41f1)
Thread T82 (rpc reactor-757) created by T0 here:
#0 0x1a94900 in __interceptor_pthread_create
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x233d1c2 in kudu::Thread::StartThread(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, boost::function<void ()> const&, unsigned long,
scoped_refptr<kudu::Thread>*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/thread.cc:619:15
#2 0x354cab5 in kudu::Status kudu::Thread::Create<void
(kudu::rpc::ReactorThread::*)(),
kudu::rpc::ReactorThread*>(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread*
const&, scoped_refptr<kudu::Thread>*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/thread.h:164:12
#3 0x3542748 in kudu::rpc::ReactorThread::Init()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:188:10
#4 0x354a982 in kudu::rpc::Reactor::Init()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:762:18
#5 0x35300cb in kudu::rpc::Messenger::Init()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:447:5
#6 0x352f77e in
kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:203:3
#7 0x241b271 in impala::RpcMgr::Init(impala::TNetworkAddress const&)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:151:3
#8 0x24859a5 in impala::ExecEnv::Init()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/exec-env.cc:408:3
#9 0x2885240 in ImpaladMain(int, char**)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impalad-main.cc:71:3
#10 0x1b736a8 in main
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
#11 0x7ff5486e9c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
Thread T112 created by T0 here:
#0 0x1a94900 in __interceptor_pthread_create
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
#1 0x44c3248 in boost::thread::start_thread_noexcept()
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x44c3248)
#2 0x2caa446 in boost::thread::thread<void
(*)(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, boost::function<void
()>, impala::ThreadDebugInfo const*, impala::Promise<long,
(impala::PromiseMode)0>*), std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
>, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long,
(impala::PromiseMode)0>*>(void (*)(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
impala::Promise<long, (impala::PromiseMode)0>*),
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
>, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*,
impala::Promise<long, (impala::PromiseMode)0>*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13
#3 0x2ca5947 in
impala::Thread::StartThread(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread,
std::default_delete<impala::Thread> >*, bool)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.cc:317:13
#4 0x2439d37 in impala::Status impala::Thread::Create<void
(impala::ImpalaServicePool::*)(),
impala::ImpalaServicePool*>(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, void (impala::ImpalaServicePool::* const&)(),
impala::ImpalaServicePool* const&, std::unique_ptr<impala::Thread,
std::default_delete<impala::Thread> >*, bool)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.h:81:12
#5 0x24357ee in impala::ImpalaServicePool::Init(int)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:92:5
#6 0x241bc5a in impala::RpcMgr::RegisterService(int, int,
kudu::rpc::GeneratedServiceIf*, impala::MemTracker*)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:163:3
#7 0x28449d7 in impala::DataStreamService::Init()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/data-stream-service.cc:79:3
#8 0x2485ada in impala::ExecEnv::Init()
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/exec-env.cc:412:3
#9 0x2885240 in ImpaladMain(int, char**)
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impalad-main.cc:71:3
#10 0x1b736a8 in main
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
#11 0x7ff5486e9c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
SUMMARY: AddressSanitizer: heap-use-after-free
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, unsigned
long)
Shadow bytes around the buggy address:
0x0ffe01ae1ab0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ffe01ae1ac0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ffe01ae1ad0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ffe01ae1ae0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ffe01ae1af0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0ffe01ae1b00:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0ffe01ae1b10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0ffe01ae1b20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0ffe01ae1b30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0ffe01ae1b40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0ffe01ae1b50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==7121==ABORTING{code}
The space is previously allocated by a thread doing
{{Coordinator::FilterState::ApplyUpdate}}. So it seems a bug in aggregating
runtime filters.
cc [~fangyurao], [~tmarshall]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
