[
https://issues.apache.org/jira/browse/HIVE-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth J updated HIVE-6326:
-----------------------------
Attachment: HIVE-6326.4.patch
Incorporated [~owen.omalley] review comments.
> Split generation in ORC may generate wrong split boundaries because of
> unaccounted padded bytes
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-6326
> URL: https://issues.apache.org/jira/browse/HIVE-6326
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.13.0
> Reporter: Prasanth J
> Assignee: Prasanth J
> Labels: orcfile
> Attachments: HIVE-6326.1.patch, HIVE-6326.2.patch, HIVE-6326.3.patch,
> HIVE-6326.4.patch
>
>
> HIVE-5091 added padding to ORC files to avoid ORC stripes straddling HDFS
> blocks. The length of this padded bytes are not stored in stripe information.
> OrcInputFormat.getSplits() uses stripeInformation.getLength() for split
> computation. stripeInformation.getLength() is sum of index length, data
> length and stripe footer length. It does not account for the length of padded
> bytes which may result in wrong split boundary.
> The fix for this is to use the offset of next stripe as the length of current
> stripe which includes the padded bytes as well.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
