[
https://issues.apache.org/jira/browse/HIVE-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900057#action_12900057
]
Ning Zhang commented on HIVE-1510:
----------------------------------
As discussed offline with Yongqiang, we should clean up the pathToPartitionInfo
to contain only canonical representations for each partition. This could result
in much cleaner code. If we do that IOPrepareCache is not needed at all and the
function getPartitionDescFromPath is just simple hash lookup. We can make it as
a follow up JIRA along with cleaning up the unnecessary info in
pathToPartitionInfo as well.
Here's some comments on the current patch:
- the IOPrepareCache is cleared in Driver, which should only contain generic
code irrespect to task types. Can you do it in ExecDriver.execute()? This will
new cache is only used in ExecDriver anyways.
- some comments on why you need a new hash map keyed with the paths only will
be helpful.
> HiveCombineInputFormat should not use prefix matching to find the
> partitionDesc for a given path
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-1510
> URL: https://issues.apache.org/jira/browse/HIVE-1510
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: He Yongqiang
> Assignee: He Yongqiang
> Attachments: hive-1510.1.patch, hive-1510.3.patch
>
>
> set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
> drop table combine_3_srcpart_seq_rc;
> create table combine_3_srcpart_seq_rc (key int , value string) partitioned by
> (ds string, hr string) stored as sequencefile;
> insert overwrite table combine_3_srcpart_seq_rc partition (ds="2010-08-03",
> hr="00") select * from src;
> alter table combine_3_srcpart_seq_rc set fileformat rcfile;
> insert overwrite table combine_3_srcpart_seq_rc partition (ds="2010-08-03",
> hr="001") select * from src;
> desc extended combine_3_srcpart_seq_rc partition(ds="2010-08-03", hr="00");
> desc extended combine_3_srcpart_seq_rc partition(ds="2010-08-03", hr="001");
> select * from combine_3_srcpart_seq_rc where ds="2010-08-03" order by key;
> drop table combine_3_srcpart_seq_rc;
> will fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
