[ https://issues.apache.org/jira/browse/IMPALA-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-9407. ----------------------------------- Resolution: Duplicate This looks like our old friend IMPALA-4631. This happened on a CentOS 6.4 vm on a m2-4xlarge ec2 instance. {noformat} F0217 02:10:47.921636 14885 runtime-profile-counters.h:572] 9945dfcfab2eb54c:1609788600000001] Check failed: offset_ >= -3 (-15 vs. -3) {noformat} > Impalad crash in RuntimeProfile::EventSequence > ---------------------------------------------- > > Key: IMPALA-9407 > URL: https://issues.apache.org/jira/browse/IMPALA-9407 > Project: IMPALA > Issue Type: Bug > Affects Versions: Impala 3.4.0 > Reporter: Attila Jeges > Assignee: Tim Armstrong > Priority: Blocker > Labels: broken-build, flaky > > impalad instance crashed during exhaustive tests, probably while running > TestExprs.test_exprs: > {code:java} > 18:00:32 TestExprs.test_exprs[protocol: beeswax | exec_option: > {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, > 'disable_codegen': False, 'abort_on_error': 1, > 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block | > enable_expr_rewrites: 0] > 18:00:32 [gw1] linux2 -- Python 2.6.6 > /data/jenkins/workspace/impala-asf-master-exhaustive-centos6-shard2/repos/Impala/bin/../infra/python/env/bin/python > 18:00:32 query_test/test_exprs.py:60: in test_exprs > 18:00:32 self.run_test_case('QueryTest/exprs', vector) > 18:00:32 common/impala_test_suite.py:659: in run_test_case > 18:00:32 result = exec_fn(query, user=test_section.get('USER', > '').strip() or None) > 18:00:32 common/impala_test_suite.py:594: in __exec_in_impala > 18:00:32 result = self.__execute_query(target_impalad_client, query, > user=user) > 18:00:32 common/impala_test_suite.py:933: in __execute_query > 18:00:32 return impalad_client.execute(query, user=user) > 18:00:32 common/impala_connection.py:205: in execute > 18:00:32 return self.__beeswax_client.execute(sql_stmt, user=user) > 18:00:32 beeswax/impala_beeswax.py:187: in execute > 18:00:32 handle = self.__execute_query(query_string.strip(), user=user) > 18:00:32 beeswax/impala_beeswax.py:365: in __execute_query > 18:00:32 self.wait_for_finished(handle) > 18:00:32 beeswax/impala_beeswax.py:386: in wait_for_finished > 18:00:32 raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > 18:00:32 E ImpalaBeeswaxException: ImpalaBeeswaxException: > 18:00:32 E Query aborted:Failed due to unreachable impalad(s): > shared-centos64-ec2-m2-4xlarge-ondemand-13c1.vpc.cloudera.com:22002 > {code} > impalad minidump: > {code:java} > Operating system: Linux > 0.0.0 Linux 2.6.32-358.14.1.el6.centos.plus.x86_64 #1 SMP > Tue Jul 16 21:33:24 UTC 2013 x86_64 > CPU: amd64 > family 6 model 45 stepping 7 > 8 CPUs > GPU: UNKNOWN > Crash reason: SIGABRT > Crash address: 0x45200004ed0 > Process uptime: not available > Thread 373 (crashed) > 0 libc-2.12.so + 0x328e5 > rax = 0x0000000000000000 rdx = 0x0000000000000006 > rcx = 0xffffffffffffffff rbx = 0x00000000072153c0 > rsi = 0x0000000000003a25 rdi = 0x0000000000004ed0 > rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a78 > r8 = 0x00007f2b804a9b00 r9 = 0x0000000000003a25 > r10 = 0x0000000000000008 r11 = 0x0000000000000202 > r12 = 0x0000000007215440 r13 = 0x000000000000004c > r14 = 0x000000000721cd84 r15 = 0x00000000072153c0 > rip = 0x00000034f4c328e5 > Found by: given as instruction pointer in context > 1 libc-2.12.so + 0x340c5 > rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a80 > rip = 0x00000034f4c340c5 > Found by: stack scanning > 2 > impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, > void>::invoke [function_template.hpp : 153 + 0xb] > rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a88 > rip = 0x00000000021b94da > Found by: stack scanning > 3 impalad!google::LogMessage::Flush() + 0x157 > rbp = 0x00007f2c83fb27a0 rsp = 0x00007f2b804a9df0 > rip = 0x0000000004e4e3b7 > Found by: call frame info > 4 impalad!google::LogMessageFatal::~LogMessageFatal() + 0xe > rbx = 0x00007f2b804a9eb0 rbp = 0x00007f2b804a9ed0 > rsp = 0x00007f2b804a9e70 r12 = 0x0000000000000001 > r13 = 0x0000000000000001 r14 = 0x0000000000000001 > r15 = 0x0000000000000001 rip = 0x0000000004e5197e > Found by: call frame info > 5 impalad!impala::RuntimeProfile::EventSequence::Start(long) > [runtime-profile-counters.h : 572 + 0xc] > rbx = 0x000000000caaee00 rbp = 0x00007f2b804a9ed0 > rsp = 0x00007f2b804a9e90 r12 = 0x0000000000000001 > r13 = 0x0000000000000001 r14 = 0x0000000000000001 > r15 = 0x0000000000000001 rip = 0x00000000021aa49c > Found by: call frame info > 6 impalad!impala::FragmentInstanceState::Prepare() > [fragment-instance-state.cc : 153 + 0x2e] > rbx = 0x000000000caaee00 rbp = 0x00007f2b804aa4f0 > rsp = 0x00007f2b804a9ee0 r12 = 0x0000000000000001 > r13 = 0x0000000000000001 r14 = 0x0000000000000001 > r15 = 0x0000000000000001 rip = 0x00000000021a42f8 > Found by: call frame info > 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc > : 75 + 0x19] > rbx = 0x0000000000000001 rbp = 0x00007f2b804aa650 > rsp = 0x00007f2b804aa500 r12 = 0x0000000000000001 > r13 = 0x0000000000000001 r14 = 0x0000000000000001 > r15 = 0x0000000000000001 rip = 0x00000000021a366e > Found by: call frame info > 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > [query-state.cc : 694 + 0x19] > rbx = 0x0000000000000001 rbp = 0x00007f2b804aa760 > rsp = 0x00007f2b804aa660 r12 = 0x0000000000000001 > r13 = 0x0000000000000001 r14 = 0x0000000000000001 > r15 = 0x0000000000000001 rip = 0x00000000021b78f3 > Found by: call frame info > 9 impalad!impala::QueryState::StartFInstances()::{lambda()#1}::operator()() > const + 0x26 > rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa780 > rsp = 0x00007f2b804aa770 r12 = 0x0000000000000000 > r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 > r15 = 0x0000000000000007 rip = 0x00000000021b5c12 > Found by: call frame info > 10 > impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, > void>::invoke [function_template.hpp : 153 + 0xc] > rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa7b0 > rsp = 0x00007f2b804aa790 r12 = 0x0000000000000000 > r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 > r15 = 0x0000000000000007 rip = 0x00000000021b94db > Found by: call frame info > 11 impalad!boost::function0<void>::operator()() const [function_template.hpp > : 767 + 0x11] > rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa7f0 > rsp = 0x00007f2b804aa7c0 r12 = 0x0000000000000000 > r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 > r15 = 0x0000000000000007 rip = 0x0000000001f9c39e > Found by: call frame info > 12 impalad!impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) [thread.cc : 360 + 0xf] > rbx = 0x0000000000003a25 rbp = 0x00007f2b804aab50 > rsp = 0x00007f2b804aa800 r12 = 0x0000000000000000 > r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 > r15 = 0x0000000000000007 rip = 0x000000000255ac29 > Found by: call frame info > 13 impalad!void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > [bind.hpp : 525 + 0x15] > rbx = 0x000000000255a8c2 rbp = 0x00007f2b804aabc0 > rsp = 0x00007f2b804aab60 r12 = 0x00007f2b80eab830 > r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8 > r15 = 0x0000000000000007 rip = 0x0000000002562fad > Found by: call frame info > 14 impalad!boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() [bind_template.hpp : 20 + 0x22] > rbx = 0x0000000000000000 rbp = 0x00007f2b804aac10 > rsp = 0x00007f2b804aabd0 r12 = 0x00000034f5011200 > r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 > r15 = 0x0000000000000007 rip = 0x0000000002562ed1 > Found by: call frame info > 15 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() [thread.hpp : 116 + 0x12] > rbx = 0x0000000000000000 rbp = 0x00007f2b804aac30 > rsp = 0x00007f2b804aac20 r12 = 0x00000034f5011200 > r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 > r15 = 0x0000000000000007 rip = 0x0000000002562e94 > Found by: call frame info > 16 impalad!thread_proxy + 0xda > rbx = 0x0000000000000000 rbp = 0x0000000000000000 > rsp = 0x00007f2b804aac40 r12 = 0x00000034f5011200 > r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 > r15 = 0x0000000000000007 rip = 0x0000000003d77d5a > Found by: call frame info > 17 libpthread-2.12.so + 0x7851 > rbx = 0x0000000000000000 rbp = 0x0000000000000000 > rsp = 0x00007f2b804aac80 r12 = 0x00000034f5011200 > r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004 > r15 = 0x0000000000000007 rip = 0x00000034f5007851 > Found by: call frame info > 18 libpthread-2.12.so + 0x11200 > rsp = 0x00007f2b804aad28 rip = 0x00000034f5011200 > Found by: stack scanning > 19 libpthread-2.12.so + 0x11200 > rsp = 0x00007f2b804aada0 rip = 0x00000034f5011200 > Found by: stack scanning > 20 libc-2.12.so + 0xe894d > rsp = 0x00007f2b804aadc0 rip = 0x00000034f4ce894d > Found by: stack scanning > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org