[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3604/ Tes

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21416 I just found that PySpark's `isin` is capable of similar feature. One issue is Scala/Java API now doesn't match with PySpark API. --- ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91178/testReport)** for PR 21416 at commit [`286a468`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91178/ Test PASSed. ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 @cloud-fan Let me know if the new API looks good to you. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21416 @viirya good point. One thing I'm not sure is, does `isin(collection: Iterable)` conflict with `isin(list: Any*)`? if they don't conflict, they we can follow pyspark. --- --

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21416 Yes. The design of PySpark API seems to be a bit different than Scala/Java API at beginning. If we are going to make them consistent, either we break Scala queries like `col.isin(Array[Byte](1,2,3))`

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 @cloud-fan unfortunately, scala vararg can not be overloaded, and scala will return the following error. ```scala Error:(410, 32) ambiguous reference to overloaded definition, both m

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91212/testReport)** for PR 21416 at commit [`1332406`](https://github.com/apache/spark/commit/13

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3629/ Tes

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21416 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apa

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21416 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91212/testReport)** for PR 21416 at commit [`1332406`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91212/ Test FAILed. ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h..

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91220/testReport)** for PR 21416 at commit [`1332406`](https://github.com/apache/spark/commit/13

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3637/ Tes

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91220 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91220/testReport)** for PR 21416 at commit [`1332406`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91220/ Test PASSed. ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21416 +1, LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91242/testReport)** for PR 21416 at commit [`fed2846`](https://github.com/apache/spark/commit/fe

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 @rxin I simplified the test cases as you suggested. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3656/ Tes

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-28 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21416 LGTM (I didn't look that carefully though) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91242/testReport)** for PR 21416 at commit [`fed2846`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91242/ Test FAILed. ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21416 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91254/testReport)** for PR 21416 at commit [`fed2846`](https://github.com/apache/spark/commit/fe

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3667/ Tes

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21416 **[Test build #91254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91254/testReport)** for PR 21416 at commit [`fed2846`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91254/ Test PASSed. ---

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21416 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21416: [SPARK-24371] [SQL] Added isInCollection in DataFrame AP...

2018-05-29 Thread dbtsai
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21416 Merged into master. Thank you everyone for reviewing. Followup PR will be created for 1. Adding tests in Java. 2. Adding docs about automagical type casting. --- --