[jira] [Comment Edited] (SPARK-17495) Hive hash implementation
[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059890#comment-17059890 ] Takeshi Yamamuro edited comment on SPARK-17495 at 3/16/20, 2:00 AM: Ah, I see. I've looked over the merged PRs and, in the branch-2.4 (the master, too), it seems the hive hash function is not used yet in partitioning/bucketing. So, we need more work to use the hash there. This Jira just targets the implementation of the hive-simulated hash func, so I think we can close this now. was (Author: maropu): Ah, I see. I've looked over the merged PRs and, in the branch-2.4 (the master, too), it seems the hive hash function is not used yet in partitioning/bucketing. So, we need more work to use the hash there. This Jira just targets the implementation of the hive-simulated hash, so I think we can close this now. > Hive hash implementation > > > Key: SPARK-17495 > URL: https://issues.apache.org/jira/browse/SPARK-17495 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.2.0 >Reporter: Tejas Patil >Assignee: Tejas Patil >Priority: Minor > Fix For: 2.2.0 > > > Spark internally uses Murmur3Hash for partitioning. This is different from > the one used by Hive. For queries which use bucketing this leads to different > results if one tries the same query on both engines. For us, we want users to > have backward compatibility to that one can switch parts of applications > across the engines without observing regressions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-17495) Hive hash implementation
[
https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059844#comment-17059844
]
Takeshi Yamamuro edited comment on SPARK-17495 at 3/16/20, 1:07 AM:
I checked the commit logs;
{code:java}
$git checkout v2.1.0
$git log --pretty=oneline | grep SPARK-17495
a99743d053e84f695dc3034550939555297b0a05 [SPARK-17495][SQL] Add Hash capability
semantically equivalent to Hive's
$git checkout v2.2.0
$git log --pretty=oneline | grep SPARK-17495
9456688547522a62f1e7520e9b3564550c57aa5d [SPARK-17495][SQL] Support date,
timestamp and interval types in Hive hash
2a0bc867a4a1dad4ecac47701199e540d345ff4f [SPARK-17495][SQL] Support Decimal
type in Hive-hash
3e40f6c3d6fc0bcd828d09031fa3994925394889 [SPARK-17495][SQL] Add more tests for
hive hash
a99743d053e84f695dc3034550939555297b0a05 [SPARK-17495][SQL] Add Hash capability
semantically equivalent to Hive's
{code}
I think the essential parts to fix this issue had been merged in branch-2.2
(the v2.2.0 release), so I will close this again with `Fix Version/s`=2.2.0. If
there is any problem, please fix this. [~dongjoon] [~hyukjin.kwon] [~viirya]
[~tejasp] l If you have more work on this, please open a new jira.
[~FelixKJose] Yea, if you use spark-2.4.x, you've already used this fix.
was (Author: maropu):
I checked the commit logs;
{code:java}
$git checkout v2.1.0
$git log --pretty=oneline | grep SPARK-17495
a99743d053e84f695dc3034550939555297b0a05 [SPARK-17495][SQL] Add Hash capability
semantically equivalent to Hive's
$git checkout v2.2.0
$git log --pretty=oneline | grep SPARK-17495
9456688547522a62f1e7520e9b3564550c57aa5d [SPARK-17495][SQL] Support date,
timestamp and interval types in Hive hash
2a0bc867a4a1dad4ecac47701199e540d345ff4f [SPARK-17495][SQL] Support Decimal
type in Hive-hash
3e40f6c3d6fc0bcd828d09031fa3994925394889 [SPARK-17495][SQL] Add more tests for
hive hash
a99743d053e84f695dc3034550939555297b0a05 [SPARK-17495][SQL] Add Hash capability
semantically equivalent to Hive's
{code}
I think the essential parts to fix this issue had been merged in branch-2.2
(the v2.2.0 release), so I will close this again with `Fix Version/s`=2.2.0. If
there is any problem, please fix this. [~dongjoon] [~hyukjin.kwon] [~viirya]
@Tejas Patil If you have more work on this, please open a new jira.
[~FelixKJose] Yea, if you use spark-2.4.x, you've already used this fix.
> Hive hash implementation
>
>
> Key: SPARK-17495
> URL: https://issues.apache.org/jira/browse/SPARK-17495
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
> Labels: bulk-closed
>
> Spark internally uses Murmur3Hash for partitioning. This is different from
> the one used by Hive. For queries which use bucketing this leads to different
> results if one tries the same query on both engines. For us, we want users to
> have backward compatibility to that one can switch parts of applications
> across the engines without observing regressions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-17495) Hive hash implementation
[ https://issues.apache.org/jira/browse/SPARK-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883203#comment-15883203 ] Tejas Patil edited comment on SPARK-17495 at 2/24/17 5:57 PM: -- [~rxin] : No probs. Any opinion about my comment from yesterday ? ie. https://issues.apache.org/jira/browse/SPARK-17495?focusedCommentId=15882161=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15882161 was (Author: tejasp): [~rxin] : No probs. Any opinion about my comment from yesterday ? > Hive hash implementation > > > Key: SPARK-17495 > URL: https://issues.apache.org/jira/browse/SPARK-17495 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Tejas Patil >Assignee: Tejas Patil >Priority: Minor > Fix For: 2.2.0 > > > Spark internally uses Murmur3Hash for partitioning. This is different from > the one used by Hive. For queries which use bucketing this leads to different > results if one tries the same query on both engines. For us, we want users to > have backward compatibility to that one can switch parts of applications > across the engines without observing regressions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
