[ 
https://issues.apache.org/jira/browse/IMPALA-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223768#comment-17223768
 ] 

Joe McDonnell commented on IMPALA-9767:
---------------------------------------

Most recent symptom (from IMPALA-10307):
{noformat}
==17479==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f056b69e800 
at pc 0x000001b049e1 bp 0x7f068b959180 sp 0x7f068b958930
READ of size 1048576 at 0x7f056b69e800 thread T82 (rpc reactor-179)
    #0 0x1b049e0 in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned 
long, unsigned long) 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
    #1 0x1b1b041 in read_msghdr(void*, __sanitizer::__sanitizer_msghdr*, long) 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2781
    #2 0x1b1d923 in __interceptor_sendmsg 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2796
    #3 0x39d846c in kudu::Socket::Writev(iovec const*, int, long*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/net/socket.cc:447:3
    #4 0x35e7c95 in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/transfer.cc:227:26
    #5 0x35f0750 in kudu::rpc::Connection::WriteHandler(ev::io&, int) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/connection.cc:802:31
    #6 0x5844bc2 in ev_invoke_pending 
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x5844bc2)
    #7 0x357aabc in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:196:3
    #8 0x584826f in ev_run 
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x584826f)
    #9 0x357acb1 in kudu::rpc::ReactorThread::RunThread() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:497:9
    #10 0x358c3eb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, 
kudu::rpc::ReactorThread>, 
boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > 
>::operator()() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #11 0x235f9d6 in boost::function0<void>::operator()() const 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #12 0x235c249 in kudu::Thread::SuperviseThread(void*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/thread.cc:675:3
    #13 0x7f0edb132e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
    #14 0x7f0ed7bc934c in __clone (/lib64/libc.so.6+0xf834c)

0x7f056b69e800 is located 0 bytes inside of 1048577-byte region 
[0x7f056b69e800,0x7f056b79e801)
freed by thread T110 here:
    #0 0x1b8a9c0 in operator delete(void*) 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
    #1 0x7f0ed84cc5a9 in __gnu_cxx::new_allocator<char>::deallocate(char*, 
unsigned long) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:125
    #2 0x7f0ed84cc5a9 in std::allocator_traits<std::allocator<char> 
>::deallocate(std::allocator<char>&, char*, unsigned long) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/alloc_traits.h:462
    #3 0x7f0ed84cc5a9 in std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:226
    #4 0x7f0ed84cc5a9 in std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:302

previously allocated by thread T111 here:
    #0 0x1b89c50 in operator new(unsigned long) 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
    #1 0x1b8e93e in void std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char 
const*, char const*, std::forward_iterator_tag) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/basic_string.tcc:219:14
    #2 0x7f0ed84ce994 in void std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char 
const*>(char const*, char const*, std::__false_type) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:236
    #3 0x7f0ed84ce994 in void std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char 
const*, char const*) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:255
    #4 0x7f0ed84ce994 in std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >::basic_string(char const*, 
unsigned long, std::allocator<char> const&) 
/mnt/source/gcc/build-7.5.0/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:502
    #5 0x365ddb5 in 
impala::Coordinator::FilterState::ApplyUpdate(impala::UpdateFilterParamsPB 
const&, impala::Coordinator*, kudu::rpc::RpcContext*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1481:51
    #6 0x365ccd1 in 
impala::Coordinator::UpdateFilter(impala::UpdateFilterParamsPB const&, 
kudu::rpc::RpcContext*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1379:12
    #7 0x298bfc5 in 
impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, 
kudu::rpc::RpcContext*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/client-request-state.cc:1501:11
    #8 0x28dbce5 in 
impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, 
impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impala-server.cc:2885:19
    #9 0x286b495 in 
impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, 
impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
    #10 0x35de6e3 in std::function<void (google::protobuf::Message const*, 
google::protobuf::Message*, 
kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, 
google::protobuf::Message*, kudu::rpc::RpcContext*) const 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706:14
    #11 0x35ddb91 in 
kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
    #12 0x2456e1e in impala::ImpalaServicePool::RunThread() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:280:15
    #13 0x245fbcb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, 
impala::ImpalaServicePool>, 
boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > 
>::operator()() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #14 0x235f9d6 in boost::function0<void>::operator()() const 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #15 0x2cd92e9 in 
impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
impala::Promise<long, (impala::PromiseMode)0>*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.cc:360:3
    #16 0x2ce4728 in void 
boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > >, 
boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, 
boost::_bi::value<impala::ThreadDebugInfo*>, 
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> 
>::operator()<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&, std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, boost::function<void 
()>, impala::ThreadDebugInfo const*, impala::Promise<long, 
(impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void 
(*&)(std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&, std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, boost::function<void 
()>, impala::ThreadDebugInfo const*, impala::Promise<long, 
(impala::PromiseMode)0>*), boost::_bi::list0&, int) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
    #17 0x2ce457b in boost::_bi::bind_t<void, void 
(*)(std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&, std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, boost::function<void 
()>, impala::ThreadDebugInfo const*, impala::Promise<long, 
(impala::PromiseMode)0>*), 
boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > >, 
boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, 
boost::_bi::value<impala::ThreadDebugInfo*>, 
boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > 
>::operator()() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #18 0x44fd111 in thread_proxy 
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x44fd111)

Thread T82 (rpc reactor-179) created by T0 here:
    #0 0x1aaf370 in __interceptor_pthread_create 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
    #1 0x235b3d2 in kudu::Thread::StartThread(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, boost::function<void ()> const&, unsigned long, 
scoped_refptr<kudu::Thread>*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/thread.cc:619:15
    #2 0x35848d5 in kudu::Status kudu::Thread::Create<void 
(kudu::rpc::ReactorThread::*)(), 
kudu::rpc::ReactorThread*>(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* 
const&, scoped_refptr<kudu::Thread>*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/util/thread.h:164:12
    #3 0x357a568 in kudu::rpc::ReactorThread::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:188:10
    #4 0x35827a2 in kudu::rpc::Reactor::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:762:18
    #5 0x3567eeb in kudu::rpc::Messenger::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:447:5
    #6 0x356759e in 
kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:203:3
    #7 0x243c2b1 in impala::RpcMgr::Init(impala::TNetworkAddress const&) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:151:3
    #8 0x24a69e5 in impala::ExecEnv::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/exec-env.cc:408:3
    #9 0x28ab240 in ImpaladMain(int, char**) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impalad-main.cc:71:3
    #10 0x1b8e118 in main 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
    #11 0x7f0ed7af2c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

Thread T110 created by T0 here:
    #0 0x1aaf370 in __interceptor_pthread_create 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
    #1 0x44fc168 in boost::thread::start_thread_noexcept() 
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x44fc168)
    #2 0x2cdd246 in boost::thread::thread<void 
(*)(std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&, std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, boost::function<void 
()>, impala::ThreadDebugInfo const*, impala::Promise<long, 
(impala::PromiseMode)0>*), std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> 
>, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, 
(impala::PromiseMode)0>*>(void (*)(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
impala::Promise<long, (impala::PromiseMode)0>*), 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> 
>, std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*, 
impala::Promise<long, (impala::PromiseMode)0>*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13
    #3 0x2cd8747 in 
impala::Thread::StartThread(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, 
std::default_delete<impala::Thread> >*, bool) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.cc:317:13
    #4 0x245ad77 in impala::Status impala::Thread::Create<void 
(impala::ImpalaServicePool::*)(), 
impala::ImpalaServicePool*>(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, void (impala::ImpalaServicePool::* const&)(), 
impala::ImpalaServicePool* const&, std::unique_ptr<impala::Thread, 
std::default_delete<impala::Thread> >*, bool) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.h:81:12
    #5 0x245682e in impala::ImpalaServicePool::Init(int) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:92:5
    #6 0x243cc9a in impala::RpcMgr::RegisterService(int, int, 
kudu::rpc::GeneratedServiceIf*, impala::MemTracker*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:163:3
    #7 0x286a967 in impala::DataStreamService::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/data-stream-service.cc:79:3
    #8 0x24a6b1a in impala::ExecEnv::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/exec-env.cc:412:3
    #9 0x28ab240 in ImpaladMain(int, char**) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impalad-main.cc:71:3
    #10 0x1b8e118 in main 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
    #11 0x7f0ed7af2c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

Thread T111 created by T0 here:
    #0 0x1aaf370 in __interceptor_pthread_create 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
    #1 0x44fc168 in boost::thread::start_thread_noexcept() 
(/data0/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/build/debug/service/impalad+0x44fc168)
    #2 0x2cdd246 in boost::thread::thread<void 
(*)(std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&, std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, boost::function<void 
()>, impala::ThreadDebugInfo const*, impala::Promise<long, 
(impala::PromiseMode)0>*), std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> 
>, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, 
(impala::PromiseMode)0>*>(void (*)(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
impala::Promise<long, (impala::PromiseMode)0>*), 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> 
>, std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*, 
impala::Promise<long, (impala::PromiseMode)0>*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13
    #3 0x2cd8747 in 
impala::Thread::StartThread(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, 
std::default_delete<impala::Thread> >*, bool) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.cc:317:13
    #4 0x245ad77 in impala::Status impala::Thread::Create<void 
(impala::ImpalaServicePool::*)(), 
impala::ImpalaServicePool*>(std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > const&, 
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > 
const&, void (impala::ImpalaServicePool::* const&)(), 
impala::ImpalaServicePool* const&, std::unique_ptr<impala::Thread, 
std::default_delete<impala::Thread> >*, bool) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/util/thread.h:81:12
    #5 0x245682e in impala::ImpalaServicePool::Init(int) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:92:5
    #6 0x243cc9a in impala::RpcMgr::RegisterService(int, int, 
kudu::rpc::GeneratedServiceIf*, impala::MemTracker*) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:163:3
    #7 0x286a967 in impala::DataStreamService::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/data-stream-service.cc:79:3
    #8 0x24a6b1a in impala::ExecEnv::Init() 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/runtime/exec-env.cc:412:3
    #9 0x28ab240 in ImpaladMain(int, char**) 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/impalad-main.cc:71:3
    #10 0x1b8e118 in main 
/data/jenkins/workspace/impala-cdpd-master-staging-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
    #11 0x7f0ed7af2c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

SUMMARY: AddressSanitizer: heap-use-after-free 
/mnt/source/llvm/llvm-5.0.1.src-p3/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
 in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, unsigned 
long)
Shadow bytes around the buggy address:
  0x0fe12d6cbcb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe12d6cbcc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe12d6cbcd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe12d6cbce0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe12d6cbcf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0fe12d6cbd00:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe12d6cbd10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe12d6cbd20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe12d6cbd30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe12d6cbd40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe12d6cbd50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==17479==ABORTING{noformat}

> ASAN crash during coordinator runtime filter updates
> ----------------------------------------------------
>
>                 Key: IMPALA-9767
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9767
>             Project: IMPALA
>          Issue Type: Bug
>            Reporter: Sahil Takiar
>            Assignee: Fang-Yu Rao
>            Priority: Major
>              Labels: asan, broken-build, crash
>         Attachments: consoleFull_asan_939.txt
>
>
> ASAN crash output:
> {code:java}
> Error MessageAddress Sanitizer message detected in 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/ee_tests/impalad.ERRORStandard
>  Error==4808==ERROR: AddressSanitizer: heap-use-after-free on address 
> 0x7f6288cbe818 at pc 0x00000199f6fe bp 0x7f63c1a8b270 sp 0x7f63c1a8aa20
> READ of size 1048576 at 0x7f6288cbe818 thread T73 (rpc reactor-552)
>     #0 0x199f6fd in read_iovec(void*, __sanitizer::__sanitizer_iovec*, 
> unsigned long, unsigned long) 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
>     #1 0x19a1f57 in read_msghdr(void*, __sanitizer::__sanitizer_msghdr*, 
> long) 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2781
>     #2 0x19a46c3 in __interceptor_sendmsg 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2796
>     #3 0x372034d in kudu::Socket::Writev(iovec const*, int, long*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/net/socket.cc:447:3
>     #4 0x331c095 in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/transfer.cc:227:26
>     #5 0x3324da1 in kudu::rpc::Connection::WriteHandler(ev::io&, int) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/connection.cc:802:31
>     #6 0x52ca4e2 in ev_invoke_pending 
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x52ca4e2)
>     #7 0x32aeadc in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:196:3
>     #8 0x52cdb03 in ev_run 
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x52cdb03)
>     #9 0x32aecd1 in kudu::rpc::ReactorThread::RunThread() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:497:9
>     #10 0x32c08db in boost::_bi::bind_t<void, boost::_mfi::mf0<void, 
> kudu::rpc::ReactorThread>, 
> boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > 
> >::operator()() 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
>     #11 0x2148c26 in boost::function0<void>::operator()() const 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
>     #12 0x2144b29 in kudu::Thread::SuperviseThread(void*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:675:3
>     #13 0x7f6c0bcf4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
>     #14 0x7f6c0885834c in __clone (/lib64/libc.so.6+0xf834c)
> 0x7f6288cbe818 is located 24 bytes inside of 1052640-byte region 
> [0x7f6288cbe800,0x7f6288dbf7e0)
> freed by thread T114 here:
>     #0 0x1a773e0 in operator delete(void*) 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
>     #1 0x7f6c090faed3 in __gnu_cxx::new_allocator<char>::deallocate(char*, 
> unsigned long) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:110
>     #2 0x7f6c090faed3 in std::string::_Rep::_M_destroy(std::allocator<char> 
> const&) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:449
>     #3 0x7f6c090faed3 in std::string::_Rep::_M_dispose(std::allocator<char> 
> const&) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:249
>     #4 0x7f6c090faed3 in std::string::reserve(unsigned long) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:511
>     #5 0x2781865 in 
> impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, 
> kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1451:11
>     #6 0x26d57d5 in 
> impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, 
> impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2694:19
>     #7 0x266bd65 in 
> impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, 
> impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
>     #8 0x27a1eed in std::_Function_handler<void (google::protobuf::Message 
> const*, google::protobuf::Message*, kudu::rpc::RpcContext*), 
> impala::DataStreamServiceIf::DataStreamServiceIf(scoped_refptr<kudu::MetricEntity>
>  const&, scoped_refptr<kudu::rpc::ResultTracker> 
> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message 
> const*, google::protobuf::Message*, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2
>     #9 0x3312e70 in std::function<void (google::protobuf::Message const*, 
> google::protobuf::Message*, 
> kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, 
> google::protobuf::Message*, kudu::rpc::RpcContext*) const 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14
>     #10 0x3312231 in 
> kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
>     #11 0x22f557b in impala::ImpalaServicePool::RunThread() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
>     #12 0x22fe3fb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, 
> impala::ImpalaServicePool>, 
> boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > 
> >::operator()() 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
>     #13 0x2148c26 in boost::function0<void>::operator()() const 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
>     #14 0x2a8c116 in impala::Thread::SuperviseThread(std::string const&, 
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
>     #15 0x2a97998 in void boost::_bi::list5<boost::_bi::value<std::string>, 
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, 
> boost::_bi::value<impala::ThreadDebugInfo*>, 
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> 
> >::operator()<void (*)(std::string const&, std::string const&, 
> boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*), 
> boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, 
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
>     #16 0x2a977eb in boost::_bi::bind_t<void, void (*)(std::string const&, 
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*), 
> boost::_bi::list5<boost::_bi::value<std::string>, 
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, 
> boost::_bi::value<impala::ThreadDebugInfo*>, 
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > 
> >::operator()() 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
>     #17 0x41a8939 in thread_proxy 
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a8939)
> previously allocated by thread T109 here:
>     #0 0x1a76668 in operator new(unsigned long) 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
>     #1 0x7f6c090fa168 in __gnu_cxx::new_allocator<char>::allocate(unsigned 
> long, void const*) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104
>     #2 0x7f6c090fa168 in std::string::_Rep::_S_create(unsigned long, unsigned 
> long, std::allocator<char> const&) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:607
>     #3 0x7f6c090fbb7b in _S_construct_aux<char const*> 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1743
>     #4 0x7f6c090fbb7b in _S_construct<char const*> 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1764
>     #5 0x7f6c090fbb7b in std::basic_string<char, std::char_traits<char>, 
> std::allocator<char> >::basic_string(char const*, unsigned long, 
> std::allocator<char> const&) 
> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:207
>     #6 0x338b610 in 
> impala::Coordinator::FilterState::ApplyUpdate(impala::UpdateFilterParamsPB 
> const&, impala::Coordinator*, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1417:51
>     #7 0x338a4c0 in 
> impala::Coordinator::UpdateFilter(impala::UpdateFilterParamsPB const&, 
> kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1315:12
>     #8 0x2781865 in 
> impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, 
> kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1451:11
>     #9 0x26d57d5 in 
> impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, 
> impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2694:19
>     #10 0x266bd65 in 
> impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, 
> impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
>     #11 0x27a1eed in std::_Function_handler<void (google::protobuf::Message 
> const*, google::protobuf::Message*, kudu::rpc::RpcContext*), 
> impala::DataStreamServiceIf::DataStreamServiceIf(scoped_refptr<kudu::MetricEntity>
>  const&, scoped_refptr<kudu::rpc::ResultTracker> 
> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message 
> const*, google::protobuf::Message*, kudu::rpc::RpcContext*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2
>     #12 0x3312e70 in std::function<void (google::protobuf::Message const*, 
> google::protobuf::Message*, 
> kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, 
> google::protobuf::Message*, kudu::rpc::RpcContext*) const 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14
>     #13 0x3312231 in 
> kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
>     #14 0x22f557b in impala::ImpalaServicePool::RunThread() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
>     #15 0x22fe3fb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, 
> impala::ImpalaServicePool>, 
> boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > 
> >::operator()() 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
>     #16 0x2148c26 in boost::function0<void>::operator()() const 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
>     #17 0x2a8c116 in impala::Thread::SuperviseThread(std::string const&, 
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
>     #18 0x2a97998 in void boost::_bi::list5<boost::_bi::value<std::string>, 
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, 
> boost::_bi::value<impala::ThreadDebugInfo*>, 
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> 
> >::operator()<void (*)(std::string const&, std::string const&, 
> boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*), 
> boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, 
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
>     #19 0x2a977eb in boost::_bi::bind_t<void, void (*)(std::string const&, 
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, 
> impala::Promise<long, (impala::PromiseMode)0>*), 
> boost::_bi::list5<boost::_bi::value<std::string>, 
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, 
> boost::_bi::value<impala::ThreadDebugInfo*>, 
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > 
> >::operator()() 
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
>     #20 0x41a8939 in thread_proxy 
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a8939)
> Thread T73 (rpc reactor-552) created by T0 here:
>     #0 0x198755d in __interceptor_pthread_create 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
>     #1 0x2143d2c in kudu::Thread::StartThread(std::string const&, std::string 
> const&, boost::function<void ()> const&, unsigned long, 
> scoped_refptr<kudu::Thread>*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:619:15
>     #2 0x32b8755 in kudu::Status kudu::Thread::Create<void 
> (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::string 
> const&, std::string const&, void (kudu::rpc::ReactorThread::* const&)(), 
> kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.h:164:12
>     #3 0x32ae5a4 in kudu::rpc::ReactorThread::Init() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:188:10
>     #4 0x32b6602 in kudu::rpc::Reactor::Init() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:762:18
>     #5 0x329bffb in kudu::rpc::Messenger::Init() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:447:5
>     #6 0x329b6ab in 
> kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:203:3
>     #7 0x22dae9f in impala::RpcMgr::Init(impala::TNetworkAddress const&) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:151:3
>     #8 0x2343545 in impala::ExecEnv::Init() 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/exec-env.cc:385:3
>     #9 0x26a8422 in ImpaladMain(int, char**) 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impalad-main.cc:73:3
>     #10 0x1a7aa9d in main 
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
>     #11 0x7f6c08781c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
> Thread T114 created by T0 here:
>     #0 0x198755d in __interceptor_pthread_create 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
>     #1 0x41a7cc9 in boost::thread::start_thread_noexcept() 
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a7cc9)
>     #2 0x45e0360d  (<unknown module>)
> Thread T109 created by T0 here:
>     #0 0x198755d in __interceptor_pthread_create 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
>     #1 0x41a7cc9 in boost::thread::start_thread_noexcept() 
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a7cc9)
>     #2 0x45e0360d  (<unknown module>)
> SUMMARY: AddressSanitizer: heap-use-after-free 
> /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
>  in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, 
> unsigned long)
> Shadow bytes around the buggy address:
>   0x0fecd118fcb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0fecd118fcc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0fecd118fcd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0fecd118fce0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0fecd118fcf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0fecd118fd00: fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0fecd118fd10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0fecd118fd20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0fecd118fd30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0fecd118fd40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0fecd118fd50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
>   Addressable:           00
>   Partially addressable: 01 02 03 04 05 06 07 
>   Heap left redzone:       fa
>   Freed heap region:       fd
>   Stack left redzone:      f1
>   Stack mid redzone:       f2
>   Stack right redzone:     f3
>   Stack after return:      f5
>   Stack use after scope:   f8
>   Global redzone:          f9
>   Global init order:       f6
>   Poisoned by user:        f7
>   Container overflow:      fc
>   Array cookie:            ac
>   Intra object redzone:    bb
>   ASan internal:           fe
>   Left alloca redzone:     ca
>   Right alloca redzone:    cb
> ==4808==ABORTING {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

Reply via email to