[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171992696
  
Ah I didn't notice @felixcheung earlier comment. Thanks for clarifying. I 
guess there is nothing to do here as the JIRA rightly says this feature is 
fixed in 2.0.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171910873
  
@shivaram Just like @felixcheung commented, the ```hash``` function was 
added only in 2.0.0. So revert it from branch 1.6 will fix the broken test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171862335
  
@shivaram  in my comment above, hash was added only in 2.0.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171733072
  
@yanboliang Could you test why this doesn't with branch 1.6 ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171048146
  
sure - sounds good to me. Can you open a JIRA and cc me and @yanboliang on 
it ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171049140
  
Actually, I am not sure if we should merge this to branch 1.6 since it is a 
new feature.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171044301
  
It breaks hadoop 1 test 
(https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-1.6-test-sbt-hadoop-1.0/30/console).
 Can you take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171048826
  
let's just reuse the original jira. I will reopen it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171047941
  
looks like 2.2 tests are broken as well. How about we revert it from 1.6 
for now? If it is good, can you do the revert? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171050117
  
The SparkR module is still considered alpha, so we do include new features 
in minor updates when it is appropriate. In this case it was a small new 
function, so it seemed fine to me. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171047282
  
Hmm is `hash` not a supported function in 1.6 by any chance ? Are the other 
hadoop versions in branch-1.6 fine ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171095115
  
reverted from 1.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171085276
  
it looks like `hash` is new in 2.0
```
  /**
   * Calculates the hash code of given columns, and returns the result as a 
int column.
   *
   * @group misc_funcs
   * @since 2.0
   */
  @scala.annotation.varargs
  def hash(cols: Column*): Column = withExpr {
new Murmur3Hash(cols.map(_.expr))
  }
```


https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L1823

so shouldn't be in 1.6.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-171094258
  
@shivaram I will revert it from 1.6 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-170202531
  
LGTM. Thanks @yanboliang - Merging this to master and `branch-1.6`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10597


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yanboliang]

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/10597

[SPARK-12645] [SparkR] SparkR support hash function

Add ```hash``` function for SparkR ```DataFrame```.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark spark-12645

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10597.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10597


commit c41eb1fd364c52d9eae0469229e0eb850c03c57a
Author: Yanbo Liang 
Date:   2016-01-05T09:42:55Z

SparkR support hash function




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-168963887
  
**[Test build #48757 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48757/consoleFull)**
 for PR 10597 at commit 
[`c41eb1f`](https://github.com/apache/spark/commit/c41eb1fd364c52d9eae0469229e0eb850c03c57a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-168963978
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48757/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-168963977
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-168956700
  
**[Test build #48757 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48757/consoleFull)**
 for PR 10597 at commit 
[`c41eb1f`](https://github.com/apache/spark/commit/c41eb1fd364c52d9eae0469229e0eb850c03c57a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169067423
  
cc @sun-rui 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169213067
  
**[Test build #48820 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48820/consoleFull)**
 for PR 10597 at commit 
[`995fd06`](https://github.com/apache/spark/commit/995fd06412c25dba89e1ae70b4417839c04ef8d6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169257628
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169209593
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48814/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169209592
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169209916
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169215075
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169215076
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48820/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169215016
  
**[Test build #48820 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48820/consoleFull)**
 for PR 10597 at commit 
[`995fd06`](https://github.com/apache/spark/commit/995fd06412c25dba89e1ae70b4417839c04ef8d6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12645] [SparkR] SparkR support hash fun...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10597#issuecomment-169260865
  
looks good. no conflict with base/stats


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org