[ 
https://issues.apache.org/jira/browse/HIVE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724461#action_12724461
 ] 

Zheng Shao commented on HIVE-461:
---------------------------------

Some preliminary comments:

@hive-461-2009-06-26.patch:
1. Use ArrayList instead of Vector: the only difference is that Vector's 
methods are synchronized. We don't need that
2. Use Integer.valueOf(int i) instead of new Integer(i) because that's more 
efficient
3. RCFileInputFormat.setReadColumnIDs/setFullyReadColumns/getReadColumnIDs -> 
Can you move these methods to some utility class? Conceptually this should not 
be specific to RCFile.
4. setReadColumnIDs and getReadColumnIDs should use the same thing 
(ArrayList<Integer> instead of int[])
5. TableScanOperator.java: Vector<Integer> neededColumnIDs; should be moved to 
tableScanDesc.java (and use ArrayList<Integer>)

Have you figured out what's wrong with input20.q?


> Optimize RCFile reading by using column pruning results
> -------------------------------------------------------
>
>                 Key: HIVE-461
>                 URL: https://issues.apache.org/jira/browse/HIVE-461
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>    Affects Versions: 0.4.0
>            Reporter: Zheng Shao
>            Assignee: Yongqiang He
>         Attachments: hive-461-2009-05-26.patch, hive-461-2009-06-26.patch
>
>
> RCFile is a column-based file format introduced in HIVE-352. Column-based 
> storage has shown better compression ratio. On our internal data set (30 
> columns, most of them are short integer strings), we are seeing 
> gzip-compressed RCFile to be 20%+ smaller than gzip-compressed SequenceFile.
> RCFIle also has the potential to improve the reading efficiency a lot since 
> it compresses each column separately.
> We should integrate RCFile with the column pruning results from Hive to make 
> the reading faster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to