Attila Jeges created IMPALA-9407: ------------------------------------ Summary: Impalad crash in RuntimeProfile::EventSequence Key: IMPALA-9407 URL: https://issues.apache.org/jira/browse/IMPALA-9407 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.4.0 Reporter: Attila Jeges Assignee: Tim Armstrong
impalad instance crashed during exhaustive tests, probably while running TestExprs.test_exprs: {code:java} 18:00:32 TestExprs.test_exprs[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block | enable_expr_rewrites: 0] 18:00:32 [gw1] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-asf-master-exhaustive-centos6-shard2/repos/Impala/bin/../infra/python/env/bin/python 18:00:32 query_test/test_exprs.py:60: in test_exprs 18:00:32 self.run_test_case('QueryTest/exprs', vector) 18:00:32 common/impala_test_suite.py:659: in run_test_case 18:00:32 result = exec_fn(query, user=test_section.get('USER', '').strip() or None) 18:00:32 common/impala_test_suite.py:594: in __exec_in_impala 18:00:32 result = self.__execute_query(target_impalad_client, query, user=user) 18:00:32 common/impala_test_suite.py:933: in __execute_query 18:00:32 return impalad_client.execute(query, user=user) 18:00:32 common/impala_connection.py:205: in execute 18:00:32 return self.__beeswax_client.execute(sql_stmt, user=user) 18:00:32 beeswax/impala_beeswax.py:187: in execute 18:00:32 handle = self.__execute_query(query_string.strip(), user=user) 18:00:32 beeswax/impala_beeswax.py:365: in __execute_query 18:00:32 self.wait_for_finished(handle) 18:00:32 beeswax/impala_beeswax.py:386: in wait_for_finished 18:00:32 raise ImpalaBeeswaxException("Query aborted:" + error_log, None) 18:00:32 E ImpalaBeeswaxException: ImpalaBeeswaxException: 18:00:32 E Query aborted:Failed due to unreachable impalad(s): shared-centos64-ec2-m2-4xlarge-ondemand-13c1.vpc.cloudera.com:22002 {code} impalad minidump: {code:java} Operating system: Linux 0.0.0 Linux 2.6.32-358.14.1.el6.centos.plus.x86_64 #1 SMP Tue Jul 16 21:33:24 UTC 2013 x86_64 CPU: amd64 family 6 model 45 stepping 7 8 CPUs GPU: UNKNOWN Crash reason: SIGABRT Crash address: 0x45200004ed0 Process uptime: not available Thread 373 (crashed) 0 libc-2.12.so + 0x328e5 rax = 0x0000000000000000 rdx = 0x0000000000000006 rcx = 0xffffffffffffffff rbx = 0x00000000072153c0 rsi = 0x0000000000003a25 rdi = 0x0000000000004ed0 rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a78 r8 = 0x00007f2b804a9b00 r9 = 0x0000000000003a25 r10 = 0x0000000000000008 r11 = 0x0000000000000202 r12 = 0x0000000007215440 r13 = 0x000000000000004c r14 = 0x000000000721cd84 r15 = 0x00000000072153c0 rip = 0x00000034f4c328e5 Found by: given as instruction pointer in context 1 libc-2.12.so + 0x340c5 rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a80 rip = 0x00000034f4c340c5 Found by: stack scanning 2 impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke [function_template.hpp : 153 + 0xb] rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a88 rip = 0x00000000021b94da Found by: stack scanning 3 impalad!google::LogMessage::Flush() + 0x157 rbp = 0x00007f2c83fb27a0 rsp = 0x00007f2b804a9df0 rip = 0x0000000004e4e3b7 Found by: call frame info 4 impalad!google::LogMessageFatal::~LogMessageFatal() + 0xe rbx = 0x00007f2b804a9eb0 rbp = 0x00007f2b804a9ed0 rsp = 0x00007f2b804a9e70 r12 = 0x0000000000000001 r13 = 0x0000000000000001 r14 = 0x0000000000000001 r15 = 0x0000000000000001 rip = 0x0000000004e5197e Found by: call frame info 5 impalad!impala::RuntimeProfile::EventSequence::Start(long) [runtime-profile-counters.h : 572 + 0xc] rbx = 0x000000000caaee00 rbp = 0x00007f2b804a9ed0 rsp = 0x00007f2b804a9e90 r12 = 0x0000000000000001 r13 = 0x0000000000000001 r14 = 0x0000000000000001 r15 = 0x0000000000000001 rip = 0x00000000021aa49c Found by: call frame info 6 impalad!impala::FragmentInstanceState::Prepare() [fragment-instance-state.cc : 153 + 0x2e] rbx = 0x000000000caaee00 rbp = 0x00007f2b804aa4f0 rsp = 0x00007f2b804a9ee0 r12 = 0x0000000000000001 r13 = 0x0000000000000001 r14 = 0x0000000000000001 r15 = 0x0000000000000001 rip = 0x00000000021a42f8 Found by: call frame info 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc : 75 + 0x19] rbx = 0x0000000000000001 rbp = 0x00007f2b804aa650 rsp = 0x00007f2b804aa500 r12 = 0x0000000000000001 r13 = 0x0000000000000001 r14 = 0x0000000000000001 r15 = 0x0000000000000001 rip = 0x00000000021a366e Found by: call frame info 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) [query-state.cc : 694 + 0x19] rbx = 0x0000000000000001 rbp = 0x00007f2b804aa760 rsp = 0x00007f2b804aa660 r12 = 0x0000000000000001 r13 = 0x0000000000000001 r14 = 0x0000000000000001 r15 = 0x0000000000000001 rip = 0x00000000021b78f3 Found by: call frame info 9 impalad!impala::QueryState::StartFInstances()::{lambda()#1}::operator()() const + 0x26 rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa780 rsp = 0x00007f2b804aa770 r12 = 0x0000000000000000 r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 r15 = 0x0000000000000007 rip = 0x00000000021b5c12 Found by: call frame info 10 impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke [function_template.hpp : 153 + 0xc] rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa7b0 rsp = 0x00007f2b804aa790 r12 = 0x0000000000000000 r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 r15 = 0x0000000000000007 rip = 0x00000000021b94db Found by: call frame info 11 impalad!boost::function0<void>::operator()() const [function_template.hpp : 767 + 0x11] rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa7f0 rsp = 0x00007f2b804aa7c0 r12 = 0x0000000000000000 r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 r15 = 0x0000000000000007 rip = 0x0000000001f9c39e Found by: call frame info 12 impalad!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) [thread.cc : 360 + 0xf] rbx = 0x0000000000003a25 rbp = 0x00007f2b804aab50 rsp = 0x00007f2b804aa800 r12 = 0x0000000000000000 r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 r15 = 0x0000000000000007 rip = 0x000000000255ac29 Found by: call frame info 13 impalad!void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) [bind.hpp : 525 + 0x15] rbx = 0x000000000255a8c2 rbp = 0x00007f2b804aabc0 rsp = 0x00007f2b804aab60 r12 = 0x00007f2b80eab830 r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 r15 = 0x0000000000000007 rip = 0x0000000002562fad Found by: call frame info 14 impalad!boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() [bind_template.hpp : 20 + 0x22] rbx = 0x0000000000000000 rbp = 0x00007f2b804aac10 rsp = 0x00007f2b804aabd0 r12 = 0x00000034f5011200 r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 r15 = 0x0000000000000007 rip = 0x0000000002562ed1 Found by: call frame info 15 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() [thread.hpp : 116 + 0x12] rbx = 0x0000000000000000 rbp = 0x00007f2b804aac30 rsp = 0x00007f2b804aac20 r12 = 0x00000034f5011200 r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 r15 = 0x0000000000000007 rip = 0x0000000002562e94 Found by: call frame info 16 impalad!thread_proxy + 0xda rbx = 0x0000000000000000 rbp = 0x0000000000000000 rsp = 0x00007f2b804aac40 r12 = 0x00000034f5011200 r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 r15 = 0x0000000000000007 rip = 0x0000000003d77d5a Found by: call frame info 17 libpthread-2.12.so + 0x7851 rbx = 0x0000000000000000 rbp = 0x0000000000000000 rsp = 0x00007f2b804aac80 r12 = 0x00000034f5011200 r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 r15 = 0x0000000000000007 rip = 0x00000034f5007851 Found by: call frame info 18 libpthread-2.12.so + 0x11200 rsp = 0x00007f2b804aad28 rip = 0x00000034f5011200 Found by: stack scanning 19 libpthread-2.12.so + 0x11200 rsp = 0x00007f2b804aada0 rip = 0x00000034f5011200 Found by: stack scanning 20 libc-2.12.so + 0xe894d rsp = 0x00007f2b804aadc0 rip = 0x00000034f4ce894d Found by: stack scanning {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org