[
https://issues.apache.org/jira/browse/HIVE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Yongqiang updated HIVE-964:
------------------------------
Attachment: hive-964-2009-12-29-4.patch
Attache a new patch.
Changes include:
1) update patch against trunk code.
2) According to an offline discussion with Namit, Ning, and Ashish. This patch
uses the original groupKey in reducer as the dummy join key for follow-up map
joins. The previous patch just uses java's UUID and a random number generator
to generate the dummy join keys.
HIVE-963 introduced a RowContainer to handle skew join keys, it will serialize
the value parts into a local file in case of OOM. Right now this patch just
gets value object from RowContainer and serialize it to HDFS. The bad thing is
that, if a row is already serialized into local file, it will need to
deserialize to object and then reserialize to HDFS. It will need to serialize
this object twice and also a deserialize.
It will be better if we can directly read RowContainer's local file and write
to HDFS. This will need to let the RowContainer to serialize the key part
together with the join values. (Right now RowContainer will only serialize join
values.) It will be better if we can enable this only when skew join is
enabled. I suggest to do this in a follow up jira because the changes for
handling skew join is already very complicated (Some comments to make the code
more easy to understand: 1) 'tag' is get differently in mapper side and reducer
side, thus they are different in MapJoinOp and JoinOp 2) join's tag order is
reordered in JoinReorder, but the operator tree is not changed. 3) map join
actually does not use tag order array, but the tag order array is used in
MapJoinOp's parent CommonJoinOp.).
> handle skewed keys for a join in a separate job
> -----------------------------------------------
>
> Key: HIVE-964
> URL: https://issues.apache.org/jira/browse/HIVE-964
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
> Attachments: hive-964-2009-12-17.txt, hive-964-2009-12-28-2.patch,
> hive-964-2009-12-29-4.patch
>
>
> The skewed keys can be written to a temporary table or file, and a followup
> conditional task can be used to perform the join on those keys.
> As a first step, JDBM can be used for those keys
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.