[
https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15599977#comment-15599977
]
Tejas Patil commented on SPARK-17495:
-------------------------------------
[~rxin] : Sorry about that. In my original PR I intentionally did not introduce
any usages of hive hash function in rest of the code to keep the PR atomic.
Although, I did not intend to have the Jira closed.
There are two places that I can think of at top of my head where hive hash can
be used:
- When hash() is called as a function in the user query. I will work on this.
- When hash partitioning is done. Would it be possible to get
https://github.com/apache/spark/pull/15300 reviewed ? After thats in, it will
allow me to do this in a meaningful way (atleast that was my main objective
behind this Jira).
> Hive hash implementation
> ------------------------
>
> Key: SPARK-17495
> URL: https://issues.apache.org/jira/browse/SPARK-17495
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Fix For: 2.1.0
>
>
> Spark internally uses Murmur3Hash for partitioning. This is different from
> the one used by Hive. For queries which use bucketing this leads to different
> results if one tries the same query on both engines. For us, we want users to
> have backward compatibility to that one can switch parts of applications
> across the engines without observing regressions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
