[
https://issues.apache.org/jira/browse/TEZ-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V resolved TEZ-821.
-------------------------
Resolution: Not A Problem
> DefaultSorter errors out with IndexOutOfBoundsException on 1.6Gb sort buffer
> ----------------------------------------------------------------------------
>
> Key: TEZ-821
> URL: https://issues.apache.org/jira/browse/TEZ-821
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.3.0
> Environment: hive-tez + tpc-ds query19 (sf=100)
> Reporter: Gopal V
> Labels: defaultsorter
>
> With a larger than usual io.sort.mb (1.6Gb, which was autoconfigured from 20%
> of -Xmx), the DefaultSorter errors out with the following
> {code}
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at
> java.nio.ByteBufferAsIntBufferL.put(ByteBufferAsIntBufferL.java:131)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.collect(DefaultSorter.java:301)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.write(DefaultSorter.java:185)
> at
> org.apache.tez.runtime.library.output.OnFileSortedOutput$1.write(OnFileSortedOutput.java:78)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$KVOutputCollector.collect(TezProcessor.java:164)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:364)
> {code}
> This appears to be a negative offset because of an overflow from a quick
> inspection, as the code to set kvindex is always modulo'd by capacity.
> {code}
> kvindex = (kvindex - NMETA + kvmeta.capacity()) % kvmeta.capacity();
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
