[
https://issues.apache.org/jira/browse/IMPALA-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9449.
-----------------------------------
Fix Version/s: Impala 3.4.0
Resolution: Fixed
This wasn't a bug in Impala itself, but a dependency. I reported it and fixed
it there
https://github.com/cloudera/impala-lzo/commit/7b475e9dd27b8c070469b78117c1d232dd1ff13f
> Crash in LZO scan with fuzz test
> --------------------------------
>
> Key: IMPALA-9449
> URL: https://issues.apache.org/jira/browse/IMPALA-9449
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: broken-build, crash
> Fix For: Impala 3.4.0
>
>
> {noformat}
> at org.apache.hadoop.cF0301 08:25:39.334376 21960
> hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check
> failed: stream_->eof()
> *** Check failure stack trace: ***
> @ 0x4e7bf5c google::LogMessage::Fail()
> @ 0x4e7d801 google::LogMessage::SendToLog()
> @ 0x4e7b936 google::LogMessage::Flush()
> @ 0x4e7eefd google::LogMessageFatal::~LogMessageFatal()
> @ 0x7fcf1248c2bd impala::HdfsLzoTextScanner::ReadAndDecompressData()
> @ 0x7fcf1248a0da impala::HdfsLzoTextScanner::ReadData()
> @ 0x7fcf1248a635 impala::HdfsLzoTextScanner::FillByteBuffer()
> @ 0x269f4ae impala::HdfsTextScanner::FillByteBufferWrapper()
> @ 0x269c047 impala::HdfsTextScanner::FinishScanRange()
> @ 0x269f272 impala::HdfsTextScanner::GetNextInternal()
> @ 0x7fcf12488b0d impala::HdfsLzoTextScanner::GetNextInternal()
> @ 0x2655808 impala::HdfsScanner::ProcessSplit()
> @ 0x27c1a4b impala::HdfsScanNode::ProcessSplit()
> @ 0x27c0c20 impala::HdfsScanNode::ScannerThread()
> @ 0x27bff7d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x27c2571
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1fb39d5 boost::function0<>::operator()()
> @ 0x257402a impala::Thread::SuperviseThread()
> @ 0x257c2ae boost::_bi::list5<>::operator()<>()
> @ 0x257c1d2 boost::_bi::bind_t<>::operator()()
> @ 0x257c195 boost::detail::thread_data<>::run()
> @ 0x3da22f9 thread_proxy
> @ 0x7fcfdab10e24 start_thread
> @ 0x7fcfd767c34c __clone
> rypto.CryptoInputStream.read(CryptoInputStr
> {noformat}
> {noformat}
> CORE: ./core.1583079954.31711.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data0/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6
> #1 0x00007f231918c8e8 in abort () from /lib64/libc.so.6
> #2 0x0000000004e85504 in google::DumpStackTraceAndExit() ()
> #3 0x0000000004e7bf5d in google::LogMessage::Fail() ()
> #4 0x0000000004e7d802 in google::LogMessage::SendToLog() ()
> #5 0x0000000004e7b937 in google::LogMessage::Flush() ()
> #6 0x0000000004e7eefe in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x00007f2254b1d2be in impala::HdfsLzoTextScanner::ReadAndDecompressData
> (this=0x182062c0, pool=0xdb6cb68) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:582
> #8 0x00007f2254b1b0db in impala::HdfsLzoTextScanner::ReadData
> (this=0x182062c0, pool=0xdb6cb68) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:307
> #9 0x00007f2254b1b636 in impala::HdfsLzoTextScanner::FillByteBuffer
> (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:342
> #10 0x000000000269f4af in impala::HdfsTextScanner::FillByteBufferWrapper
> (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:476
> #11 0x000000000269c048 in impala::HdfsTextScanner::FinishScanRange
> (this=0x182062c0, row_batch=0xdb6cb40) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:282
> #12 0x000000000269f273 in impala::HdfsTextScanner::GetNextInternal
> (this=0x182062c0, row_batch=0xdb6cb40) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:467
> #13 0x00007f2254b19b0e in impala::HdfsLzoTextScanner::GetNextInternal
> (this=0x182062c0, row_batch=0xdb6cb40) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:144
> #14 0x0000000002655809 in impala::HdfsScanner::ProcessSplit (this=0x182062c0)
> at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scanner.cc:135
> #15 0x00000000027c1a4c in impala::HdfsScanNode::ProcessSplit
> (this=0xa3d78d00, filter_ctxs=..., expr_results_pool=0x7f2240920410,
> scan_range=0x109d1f180, scanner_thread_reservation=0x7f2240920368) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:515
> #16 0x00000000027c0c21 in impala::HdfsScanNode::ScannerThread
> (this=0xa3d78d00, first_thread=false, scanner_thread_reservation=8192) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:417
> #17 0x00000000027bff7e in impala::HdfsScanNode::<lambda()>::operator()(void)
> const (__closure=0x7f2240920b98) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:338
> #18 0x00000000027c2572 in
> boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
> #19 0x0000000001fb39d6 in boost::function0<void>::operator()
> (this=0x7f2240920b90) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:771
> #20 0x000000000257402b in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=...,
> functor=..., parent_thread_info=0x7f2242720840,
> thread_started=0x7f224271f540) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/util/thread.cc:360
> #21 0x000000000257c2af in boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*>
> >::operator()<void (*)(std::string const&, std::string const&,
> boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&,
> std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int)
> (this=0x1989013c0, f=@0x1989013b8: 0x2573cc4
> <impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function<void ()>, impala::ThreadDebugInfo const*,
> impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531
> #22 0x000000000257c1d3 in boost::_bi::bind_t<void, void (*)(std::string
> const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo
> const*, impala::Promise<long, (impala::PromiseMode)0>*),
> boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >
> >::operator()() (this=0x1989013b8) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222
> #23 0x000000000257c196 in boost::detail::thread_data<boost::_bi::bind_t<void,
> void (*)(std::string const&, std::string const&, boost::function<void ()>,
> impala::ThreadDebugInfo const*, impala::Promise<long,
> (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>,
> boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >,
> boost::_bi::value<impala::ThreadDebugInfo*>,
> boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >
> >::run() (this=0x198901200) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116
> #24 0x0000000003da22fa in thread_proxy ()
> #25 0x00007f231c6e2e25 in start_thread () from /lib64/libpthread.so.0
> #26 0x00007f231924e34d in clone () from /lib64/libc.so.6
> {noformat}
> This crashed all 3 impalads at around the same time:
> {noformat}
> Log file created at: 2020/03/01 08:26:07
> Running on machine:
> impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F0301 08:26:07.666946 27213 hdfs-lzo-text-scanner.cc:582]
> f44a5121256f5b7d:8e0bf92c00000001] Check failed: stream_->eof()
> Log file created at: 2020/03/01 08:25:53
> Running on machine:
> impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F0301 08:25:53.913563 25657 hdfs-lzo-text-scanner.cc:582]
> 1d4781605704837d:2a54525400000001] Check failed: stream_->eof()
> Log file created at: 2020/03/01 08:25:39
> Running on machine:
> impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582]
> 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof()
> {noformat}
> There was some noise in the log around the same time, unsure if it's relevant
> {noformat}
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
> hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
> ReadOnlyBufferException: java.nio.ReadOnlyBufferException
> at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
> at
> org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
> at org.apache.hadoop.cF0301 08:25:39.334376 21960
> hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check
> failed: stream_->eof()
> *** Check failure stack trace: ***
> {noformat}
> This was testing with a slightly old impala-lzo commit:
> https://github.com/cloudera/impala-lzo/commit/36d19a8a4559ec6703e5879684b0d341c75c0202
> I'll attach logs and more info soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
