[
https://issues.apache.org/jira/browse/ORC-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated ORC-700:
---------------------------------------
Description:
*Observation*: Seems that Tests are currently using HadoopShimsPre2_3 (cause of
min.hadoop.version) that does not support a zero copy reader – for testing
purposes we probably want to test the latest Shims (including the most
up-to-date features) – opening a jira for this.
Bumping +min.hadoop.version+ to *2.7.3* and enabling zero copy reveals an
underlying issue in buffer management for the StripePlanner:
For example running: *testReadFormat_0_11* with
+conf.setBoolean(OrcConf.USE_ZEROCOPY.getHiveConfName(), true)+
{code:java}
java.lang.IllegalArgumentException: tried to release a buffer that was not
created by this stream.
at
org.apache.hadoop.fs.FSDataInputStream.releaseBuffer(FSDataInputStream.java:217)
at
org.apache.orc.impl.ZeroCopyShims$ZeroCopyAdapter.releaseBuffer(ZeroCopyShims.java:75)
at
org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.releaseBuffer(RecordReaderUtils.java:134)
at
org.apache.orc.impl.reader.StripePlanner$StreamInformation.releaseBuffers(StripePlanner.java:518)
at
org.apache.orc.impl.reader.StripePlanner.clearStreams(StripePlanner.java:183)
at
org.apache.orc.impl.RecordReaderImpl.clearStreams(RecordReaderImpl.java:1089)
at
org.apache.orc.impl.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:1124)
at
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1098)
at
org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1348)
at
org.apache.orc.TestVectorOrcFile.testReadFormat_0_11(TestVectorOrcFile.java:341)
{code}
was:
*Observation*: Seems that Tests are currently using HadoopShimsPre2_3 (cause of
min.hadoop.version) that does not support a zero copy reader – for testing
purposes we probably want to test the latest Shims (including the most
up-to-date features) – opening a jira for this.
Bumping +min.hadoop.version+ to *2.7.4* and enabling zero copy reveals an
underlying issue in buffer management for the StripePlanner:
For example running: *testReadFormat_0_11* with
+conf.setBoolean(OrcConf.USE_ZEROCOPY.getHiveConfName(), true)+
{code:java}
java.lang.IllegalArgumentException: tried to release a buffer that was not
created by this stream.
at
org.apache.hadoop.fs.FSDataInputStream.releaseBuffer(FSDataInputStream.java:217)
at
org.apache.orc.impl.ZeroCopyShims$ZeroCopyAdapter.releaseBuffer(ZeroCopyShims.java:75)
at
org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.releaseBuffer(RecordReaderUtils.java:134)
at
org.apache.orc.impl.reader.StripePlanner$StreamInformation.releaseBuffers(StripePlanner.java:518)
at
org.apache.orc.impl.reader.StripePlanner.clearStreams(StripePlanner.java:183)
at
org.apache.orc.impl.RecordReaderImpl.clearStreams(RecordReaderImpl.java:1089)
at
org.apache.orc.impl.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:1124)
at
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1098)
at
org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1348)
at
org.apache.orc.TestVectorOrcFile.testReadFormat_0_11(TestVectorOrcFile.java:341)
{code}
> StripePlanner buffer management issues with zero copy
> -----------------------------------------------------
>
> Key: ORC-700
> URL: https://issues.apache.org/jira/browse/ORC-700
> Project: ORC
> Issue Type: Bug
> Reporter: Panagiotis Garefalakis
> Priority: Major
>
> *Observation*: Seems that Tests are currently using HadoopShimsPre2_3 (cause
> of min.hadoop.version) that does not support a zero copy reader – for testing
> purposes we probably want to test the latest Shims (including the most
> up-to-date features) – opening a jira for this.
> Bumping +min.hadoop.version+ to *2.7.3* and enabling zero copy reveals an
> underlying issue in buffer management for the StripePlanner:
> For example running: *testReadFormat_0_11* with
> +conf.setBoolean(OrcConf.USE_ZEROCOPY.getHiveConfName(), true)+
> {code:java}
> java.lang.IllegalArgumentException: tried to release a buffer that was not
> created by this stream.
> at
> org.apache.hadoop.fs.FSDataInputStream.releaseBuffer(FSDataInputStream.java:217)
> at
> org.apache.orc.impl.ZeroCopyShims$ZeroCopyAdapter.releaseBuffer(ZeroCopyShims.java:75)
> at
> org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.releaseBuffer(RecordReaderUtils.java:134)
> at
> org.apache.orc.impl.reader.StripePlanner$StreamInformation.releaseBuffers(StripePlanner.java:518)
> at
> org.apache.orc.impl.reader.StripePlanner.clearStreams(StripePlanner.java:183)
> at
> org.apache.orc.impl.RecordReaderImpl.clearStreams(RecordReaderImpl.java:1089)
> at
> org.apache.orc.impl.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:1124)
> at
> org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1098)
> at
> org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1348)
> at
> org.apache.orc.TestVectorOrcFile.testReadFormat_0_11(TestVectorOrcFile.java:341)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
