[
https://issues.apache.org/jira/browse/HIVE-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107577#comment-14107577
]
Szehon Ho commented on HIVE-7856:
---------------------------------
Some preliminary research on how this would work, we would use the SPARK-2978
group+sort in the shuffle phase, and we can use the HashPartitioner to decide
the partition. Today in MapReduce we are using the default partitioner (hash
partitioner), as the HiveKey has a pre-computed hash-code field, that directs
it to the right HiveReduce function.
One issue is that HiveBaseFunctionResultList clones the HiveKey as
BytesWritable during mapper-output generation, losing the hashcode. We need to
make a change there to preserve the hashcode, for this to work.
> Enable parallelism in Reduce Side Join [Spark Branch]
> -----------------------------------------------------
>
> Key: HIVE-7856
> URL: https://issues.apache.org/jira/browse/HIVE-7856
> Project: Hive
> Issue Type: New Feature
> Components: Spark
> Reporter: Szehon Ho
>
> This is dependent on new transformation to be provided by SPARK-2978, see
> parent JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
