[ https://issues.apache.org/jira/browse/IMPALA-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-9449: ---------------------------------- Priority: Critical (was: Blocker) > Crash in LZO scan with data cache enabled > ----------------------------------------- > > Key: IMPALA-9449 > URL: https://issues.apache.org/jira/browse/IMPALA-9449 > Project: IMPALA > Issue Type: Bug > Components: Backend > Reporter: Tim Armstrong > Priority: Critical > Labels: broken-build, crash > > {noformat} > at org.apache.hadoop.cF0301 08:25:39.334376 21960 > hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check > failed: stream_->eof() > *** Check failure stack trace: *** > @ 0x4e7bf5c google::LogMessage::Fail() > @ 0x4e7d801 google::LogMessage::SendToLog() > @ 0x4e7b936 google::LogMessage::Flush() > @ 0x4e7eefd google::LogMessageFatal::~LogMessageFatal() > @ 0x7fcf1248c2bd impala::HdfsLzoTextScanner::ReadAndDecompressData() > @ 0x7fcf1248a0da impala::HdfsLzoTextScanner::ReadData() > @ 0x7fcf1248a635 impala::HdfsLzoTextScanner::FillByteBuffer() > @ 0x269f4ae impala::HdfsTextScanner::FillByteBufferWrapper() > @ 0x269c047 impala::HdfsTextScanner::FinishScanRange() > @ 0x269f272 impala::HdfsTextScanner::GetNextInternal() > @ 0x7fcf12488b0d impala::HdfsLzoTextScanner::GetNextInternal() > @ 0x2655808 impala::HdfsScanner::ProcessSplit() > @ 0x27c1a4b impala::HdfsScanNode::ProcessSplit() > @ 0x27c0c20 impala::HdfsScanNode::ScannerThread() > @ 0x27bff7d > _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv > @ 0x27c2571 > _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE > @ 0x1fb39d5 boost::function0<>::operator()() > @ 0x257402a impala::Thread::SuperviseThread() > @ 0x257c2ae boost::_bi::list5<>::operator()<>() > @ 0x257c1d2 boost::_bi::bind_t<>::operator()() > @ 0x257c195 boost::detail::thread_data<>::run() > @ 0x3da22f9 thread_proxy > @ 0x7fcfdab10e24 start_thread > @ 0x7fcfd767c34c __clone > rypto.CryptoInputStream.read(CryptoInputStr > {noformat} > {noformat} > CORE: ./core.1583079954.31711.impalad > BINARY: ./be/build/latest/service/impalad > Core was generated by > `/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be'. > Program terminated with signal SIGABRT, Aborted. > #0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6 > To enable execution of this file add > add-auto-load-safe-path > /data0/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py > line to your configuration file "/var/lib/jenkins/.gdbinit". > To completely disable this security protection add > set auto-load safe-path / > line to your configuration file "/var/lib/jenkins/.gdbinit". > For more information about this security protection see the > "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: > info "(gdb)Auto-loading safe path" > #0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6 > #1 0x00007f231918c8e8 in abort () from /lib64/libc.so.6 > #2 0x0000000004e85504 in google::DumpStackTraceAndExit() () > #3 0x0000000004e7bf5d in google::LogMessage::Fail() () > #4 0x0000000004e7d802 in google::LogMessage::SendToLog() () > #5 0x0000000004e7b937 in google::LogMessage::Flush() () > #6 0x0000000004e7eefe in google::LogMessageFatal::~LogMessageFatal() () > #7 0x00007f2254b1d2be in impala::HdfsLzoTextScanner::ReadAndDecompressData > (this=0x182062c0, pool=0xdb6cb68) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:582 > #8 0x00007f2254b1b0db in impala::HdfsLzoTextScanner::ReadData > (this=0x182062c0, pool=0xdb6cb68) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:307 > #9 0x00007f2254b1b636 in impala::HdfsLzoTextScanner::FillByteBuffer > (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:342 > #10 0x000000000269f4af in impala::HdfsTextScanner::FillByteBufferWrapper > (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:476 > #11 0x000000000269c048 in impala::HdfsTextScanner::FinishScanRange > (this=0x182062c0, row_batch=0xdb6cb40) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:282 > #12 0x000000000269f273 in impala::HdfsTextScanner::GetNextInternal > (this=0x182062c0, row_batch=0xdb6cb40) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:467 > #13 0x00007f2254b19b0e in impala::HdfsLzoTextScanner::GetNextInternal > (this=0x182062c0, row_batch=0xdb6cb40) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:144 > #14 0x0000000002655809 in impala::HdfsScanner::ProcessSplit (this=0x182062c0) > at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scanner.cc:135 > #15 0x00000000027c1a4c in impala::HdfsScanNode::ProcessSplit > (this=0xa3d78d00, filter_ctxs=..., expr_results_pool=0x7f2240920410, > scan_range=0x109d1f180, scanner_thread_reservation=0x7f2240920368) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:515 > #16 0x00000000027c0c21 in impala::HdfsScanNode::ScannerThread > (this=0xa3d78d00, first_thread=false, scanner_thread_reservation=8192) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:417 > #17 0x00000000027bff7e in impala::HdfsScanNode::<lambda()>::operator()(void) > const (__closure=0x7f2240920b98) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:338 > #18 0x00000000027c2572 in > boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>, > void>::invoke(boost::detail::function::function_buffer &) > (function_obj_ptr=...) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159 > #19 0x0000000001fb39d6 in boost::function0<void>::operator() > (this=0x7f2240920b90) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:771 > #20 0x000000000257402b in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=..., > functor=..., parent_thread_info=0x7f2242720840, > thread_started=0x7f224271f540) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/util/thread.cc:360 > #21 0x000000000257c2af in boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > (this=0x1989013c0, f=@0x1989013b8: 0x2573cc4 > <impala::Thread::SuperviseThread(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531 > #22 0x000000000257c1d3 in boost::_bi::bind_t<void, void (*)(std::string > const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo > const*, impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() (this=0x1989013b8) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 > #23 0x000000000257c196 in boost::detail::thread_data<boost::_bi::bind_t<void, > void (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() (this=0x198901200) at > /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116 > #24 0x0000000003da22fa in thread_proxy () > #25 0x00007f231c6e2e25 in start_thread () from /lib64/libpthread.so.0 > #26 0x00007f231924e34d in clone () from /lib64/libc.so.6 > {noformat} > This crashed all 3 impalads at around the same time: > {noformat} > Log file created at: 2020/03/01 08:26:07 > Running on machine: > impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg > F0301 08:26:07.666946 27213 hdfs-lzo-text-scanner.cc:582] > f44a5121256f5b7d:8e0bf92c00000001] Check failed: stream_->eof() > Log file created at: 2020/03/01 08:25:53 > Running on machine: > impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg > F0301 08:25:53.913563 25657 hdfs-lzo-text-scanner.cc:582] > 1d4781605704837d:2a54525400000001] Check failed: stream_->eof() > Log file created at: 2020/03/01 08:25:39 > Running on machine: > impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg > F0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] > 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() > {noformat} > There was some noise in the log around the same time, unsure if it's relevant > {noformat} > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at > org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) > hadoopZeroCopyRead: ZeroCopyCursor#read failed error: > ReadOnlyBufferException: java.nio.ReadOnlyBufferException > at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) > at > org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) > at org.apache.hadoop.cF0301 08:25:39.334376 21960 > hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check > failed: stream_->eof() > *** Check failure stack trace: *** > {noformat} > This was testing with a slightly old impala-lzo commit: > https://github.com/cloudera/impala-lzo/commit/36d19a8a4559ec6703e5879684b0d341c75c0202 > I'll attach logs and more info soon. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org