[
https://issues.apache.org/jira/browse/IMPALA-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Hecht reassigned IMPALA-7033:
---------------------------------
Assignee: Dan Hecht
Fix Version/s: Impala 3.1.0
Was able to confirm the race was in the patch that was backed out by
reproducing this dcheck failure in debug builds in a similar location by
looping query_test and failure in parallel for about an hour. The bug is
described in IMPALA-5384, where I'll redo the patch with the fix.
{code:java}
#4 0x00000000042958f2 in google::LogMessage::SendToLog() ()
#5 0x0000000004293a27 in google::LogMessage::Flush() ()
#6 0x0000000004296fee in google::LogMessageFatal::~LogMessageFatal() ()
#7 0x0000000001ca2deb in impala::QueryState::AcquireExecResourceRefcount()
(this=0x2c208800) at /home/dhecht/src/Impala/be/src/runtime/query-state.cc:380
#8 0x0000000001c9fa5c in
impala::QueryState::Init(impala::TExecQueryFInstancesParams const&)
(this=0x2c208800, rpc_params=...) at
/home/dhecht/src/Impala/be/src/runtime/query-state.cc:116
#9 0x0000000001cbdf16 in
impala::QueryExecMgr::StartQuery(impala::TExecQueryFInstancesParams const&)
(this=0xc848300, params=...) at
/home/dhecht/src/Impala/be/src/runtime/query-exec-mgr.cc:51
#10 0x0000000001dd00d4 in
impala::ImpalaInternalService::ExecQueryFInstances(impala::TExecQueryFInstancesResult&,
impala::TExecQueryFInstancesParams const&) (this=0xd0e1720, return_val=...,
params=...)
at /home/dhecht/src/Impala/be/src/service/impala-internal-service.cc:51
#11 0x0000000002eb80ec in
impala::ImpalaInternalServiceProcessor::process_ExecQueryFInstances(int,
apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*,
void*) (this=0xd2f8480, seqid=0, iprot=0x17b3f000, oprot=0x194d4b00,
callContext=0x195e9140) at
/home/dhecht/src/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1523{code}
> Impala crashes on exhaustive release tests
> ------------------------------------------
>
> Key: IMPALA-7033
> URL: https://issues.apache.org/jira/browse/IMPALA-7033
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Joe McDonnell
> Assignee: Dan Hecht
> Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> Exhaustive release builds have seen crashes related to memory allocation/free:
> {noformat}
> CORE: ./core.1526387352.3540.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/bu'.
> Program terminated with signal 6, Aborted.
> #0 0x0000003f10e328e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x0000003f10e328e5 in raise () from /lib64/libc.so.6
> #1 0x0000003f10e340c5 in abort () from /lib64/libc.so.6
> #2 0x00007f7d7ff261a5 in os::abort(bool) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #3 0x00007f7d800b6843 in VMError::report_and_die() () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #4 0x00007f7d7ff2b562 in JVM_handle_linux_signal () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #5 0x00007f7d7ff224f3 in signalHandler(int, siginfo*, void*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6 <signal handler called>
> #7 0x00000000026be93f in tc_newarray ()
> #8 0x0000000000c9d508 in allocate (this=0x11f1cae8, __n=1) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/ext/new_allocator.h:104
> #9 allocate (this=0x11f1cae8, __n=1) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/alloc_traits.h:357
> #10 _M_allocate (this=0x11f1cae8, __n=1) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:170
> #11 std::vector<int, std::allocator<int> >::_M_default_append
> (this=0x11f1cae8, __n=1) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/vector.tcc:557
> #12 0x0000000000cb035f in _M_default_append (this=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_map.h:506
> #13 resize (this=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:676
> #14 impala::THdfsScanNode::read (this=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/PlanNodes_types.cpp:1544
> #15 0x0000000000cb3148 in impala::TPlanNode::read (this=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/PlanNodes_types.cpp:5302
> #16 0x0000000000cbc206 in impala::TPlan::read (this=0x235e1808,
> iprot=0x901c000) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/PlanNodes_types.cpp:5871
> #17 0x0000000000ccfc0d in impala::TPlanFragment::read (this=0x235e17f0,
> iprot=0x901c000) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/Planner_types.cpp:110
> #18 0x0000000000b5069f in impala::TPlanFragmentCtx::read (this=0x235e17e8,
> iprot=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp:2849
> #19 0x0000000000b52461 in impala::TExecQueryFInstancesParams::read
> (this=0x7f7cd4b572e8, iprot=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService_types.cpp:3534
> #20 0x00000000012f63ed in
> impala::ImpalaInternalService_ExecQueryFInstances_args::read
> (this=0x7f7cd4b572e0, iprot=0x901c000) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:39
> #21 0x00000000012f6e77 in
> impala::ImpalaInternalServiceProcessor::process_ExecQueryFInstances
> (this=0xbc19e60, seqid=0, iprot=0x901c000, oprot=0x901c040, callContext=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1513
> #22 0x00000000012f601d in
> impala::ImpalaInternalServiceProcessor::dispatchCall (this=0xbc19e60,
> iprot=0x901c000, oprot=0x901c040, fname=..., seqid=0, callContext=0x901c180)
> at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1496
> #23 0x0000000000a3793a in apache::thrift::TDispatchProcessor::process
> (this=0xbc19e60, in=..., out=..., connectionContext=0x901c180) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p4/include/thrift/TDispatchProcessor.h:121
> #24 0x0000000000d762f9 in
> apache::thrift::server::TAcceptQueueServer::Task::run (this=0xbc67380) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:68
> #25 0x0000000000d6e4f9 in impala::ThriftThread::RunRunnable (this=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/rpc/thrift-thread.cc:74
> #26 0x0000000000d6f372 in operator() (function_obj_ptr=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/mem_fn_template.hpp:280
> #27 operator()<boost::_mfi::mf2<void, impala::ThriftThread,
> boost::shared_ptr<apache::thrift::concurrency::Runnable>,
> impala::Promise<long unsigned int>*>, boost::_bi::list0> (function_obj_ptr=)
> at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:392
> #28 operator() (function_obj_ptr=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #29
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf2<void, impala::ThriftThread,
> boost::shared_ptr<apache::thrift::concurrency::Runnable>,
> impala::Promise<unsigned long>*>,
> boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>,
> boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >,
> boost::_bi::value<impala::Promise<unsigned long>*> > >, void>::invoke
> (function_obj_ptr=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #30 0x0000000000f9c28f in operator() (name=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #31 impala::Thread::SuperviseThread (name=) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/util/thread.cc:356
> #32 0x0000000000f9cd8a in operator()<void (*)(const std::basic_string<char>&,
> const std::basic_string<char>&, boost::function<void()>, const
> impala::ThreadDebugInfo*, impala::Promise<long int>*), boost::_bi::list0>
> (this=0xa803400) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #33 operator() (this=0xa803400) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #34 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >&,
> const std::basic_string<char, std::char_traits<char>, std::allocator<char>
> >&, boost::function<void()>, const impala::ThreadDebugInfo*,
> impala::Promise<long int>*),
> boost::_bi::list5<boost::_bi::value<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long int>*> > > >::run(void)
> (this=0xa803400) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #35 0x000000000153f8fa in thread_proxy ()
> #36 0x0000003f11207851 in start_thread () from /lib64/libpthread.so.0
> #37 0x0000003f10ee894d in clone () from /lib64/libc.so.6{noformat}
> Additionally:
> {noformat}
> CORE: ./core.1526214223.23025.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/bu'.
> Program terminated with signal 6, Aborted.
> #0 0x00000034a38328e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x00000034a38328e5 in raise () from /lib64/libc.so.6
> #1 0x00000034a38340c5 in abort () from /lib64/libc.so.6
> #2 0x000000000260c42d in tcmalloc::Log(tcmalloc::LogMode, char const*, int,
> tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) ()
> #3 0x000000000260615d in (anonymous namespace)::do_free_helper ()
> #4 0x00000000026bcce8 in tc_free ()
> #5 0x00007f85e5690279 in os::free(void*, unsigned short) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6 0x00007f85e5800f4b in Unsafe_FreeMemory () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #7 0x00007f85cba9ab68 in ?? ()
> #8 0x000000039c676880 in ?? ()
> #9 0x00000001e9003bc0 in ?? ()
> #10 0x00007f852148eca0 in ?? ()
> #11 0x000000000260a40c in tcmalloc::CentralFreeList::InsertRange(void*,
> void*, int) ()
> #12 0x00007f85cb21b360 in ?? ()
> #13 0x0000000003d4d240 in ?? ()
> #14 0x00007f85cb21b360 in ?? ()
> #15 0x006c006200750070 in ?? ()
> #16 0x00007f85e57af13d in AbstractInterpreter::can_be_compiled(methodHandle)
> () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #17 0x00007f85cb2144e7 in ?? ()
> #18 0x000000065fe86af8 in ?? ()
> #19 0x0000000000001fa1 in ?? () at
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/uuid/sha1.hpp:139
> #20 0x0000000009740590 in ?? ()
> #21 0x0000000012027000 in ?? ()
> #22 0x00007f852148ee10 in ?? ()
> #23 0x000000065fe86af8 in ?? ()
> #24 0x00007f852148edd0 in ?? ()
> #25 0x00007f852148edd0 in ?? ()
> #26 0x00007f852148f108 in ?? ()
> #27 0x000000060000000a in ?? ()
> #28 0x00007f859f00bb08 in ?? ()
> #29 0x00007f85cb2292e0 in ?? ()
> #30 0x00007f852148ef48 in ?? ()
> #31 0x00007f852148ef00 in ?? ()
> #32 0x00007f85e5420356 in JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> Backtrace stopped: frame did not save the PC{noformat}
> Not seen on 2.x yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
