[
https://issues.apache.org/jira/browse/HIVE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725316#action_12725316
]
Zheng Shao commented on HIVE-461:
---------------------------------
@hive-461-2009-06-27.patch:
Can you do 3 things:
1. Add a comment in TableSinkOperator about the reason that you mentioned.
2. Add a test case where we join 2 RCFile tables, one of them using the first
column only, one of them using the second column only. We want to make sure
that reading heterogeneous input files works with RCFile. (Basically verify the
code in HiveInputFormat that sets the option)
3. Regenerate the patch since there are conflicts.
> Optimize RCFile reading by using column pruning results
> -------------------------------------------------------
>
> Key: HIVE-461
> URL: https://issues.apache.org/jira/browse/HIVE-461
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.4.0
> Reporter: Zheng Shao
> Assignee: Yongqiang He
> Attachments: hive-461-2009-05-26.patch, hive-461-2009-06-26.patch,
> hive-461-2009-06-27.patch
>
>
> RCFile is a column-based file format introduced in HIVE-352. Column-based
> storage has shown better compression ratio. On our internal data set (30
> columns, most of them are short integer strings), we are seeing
> gzip-compressed RCFile to be 20%+ smaller than gzip-compressed SequenceFile.
> RCFIle also has the potential to improve the reading efficiency a lot since
> it compresses each column separately.
> We should integrate RCFile with the column pruning results from Hive to make
> the reading faster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.