Panagiotis Garefalakis created ORC-700:
------------------------------------------
Summary: StripePlanner buffer management issues with zero copy
Key: ORC-700
URL: https://issues.apache.org/jira/browse/ORC-700
Project: ORC
Issue Type: Bug
Reporter: Panagiotis Garefalakis
*Observation*: Seems that Tests are currently using HadoopShimsPre2_3 (cause of
min.hadoop.version) that does not support a zero copy reader – for testing
purposes we probably want to test the latest Shims (including the most
up-to-date features) – opening a jira for this.
Bumping +min.hadoop.version+ to *2.7.4* and enabling zero copy reveals an
underlying issue in buffer management for the StripePlanner:
For example running: *testReadFormat_0_11* with
+conf.setBoolean(OrcConf.USE_ZEROCOPY.getHiveConfName(), true)+
{code:java}
java.lang.IllegalArgumentException: tried to release a buffer that was not
created by this stream.
at
org.apache.hadoop.fs.FSDataInputStream.releaseBuffer(FSDataInputStream.java:217)
at
org.apache.orc.impl.ZeroCopyShims$ZeroCopyAdapter.releaseBuffer(ZeroCopyShims.java:75)
at
org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.releaseBuffer(RecordReaderUtils.java:134)
at
org.apache.orc.impl.reader.StripePlanner$StreamInformation.releaseBuffers(StripePlanner.java:518)
at
org.apache.orc.impl.reader.StripePlanner.clearStreams(StripePlanner.java:183)
at
org.apache.orc.impl.RecordReaderImpl.clearStreams(RecordReaderImpl.java:1089)
at
org.apache.orc.impl.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:1124)
at
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1098)
at
org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1348)
at
org.apache.orc.TestVectorOrcFile.testReadFormat_0_11(TestVectorOrcFile.java:341)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
