[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6908#issuecomment-113626980 [Test build #35314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35314/consoleFull) for PR 6908 at commit [`be795e0`](https://github.com/apache/spark/commit/be795e0c4112b5e30e3387e6d1fc98b7df26c81f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6908#issuecomment-113651550 [Test build #35314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35314/console) for PR 6908 at commit [`be795e0`](https://github.com/apache/spark/commit/be795e0c4112b5e30e3387e6d1fc98b7df26c81f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6908#discussion_r32864207 --- Diff: python/pyspark/sql/column.py --- @@ -326,6 +326,27 @@ def between(self, lowerBound, upperBound): return (self = lowerBound) (self = upperBound) +@since(1.5) +def In(self, *values): + +A boolean expression that is evaluated to true if the value of this +expression is any of the given columns. +NOTE: Normally, we shold name this function the small case `in`. However, `in` is +a reserved word in Python. So we can't help naming this the upper case `In`. + + df.select(df.name, df.age, df.age.In(2, 4)).show() ++-+---+-+ +| name|age|(age = 2)| ++-+---+-+ +|Alice| 2| true| +| Bob| 5|false| ++-+---+-+ + +for v in values: --- End diff -- This approach will not scale if you have many values, please call the java API `in`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user yu-iskw commented on a diff in the pull request: https://github.com/apache/spark/pull/6908#discussion_r32864408 --- Diff: python/pyspark/sql/column.py --- @@ -326,6 +326,27 @@ def between(self, lowerBound, upperBound): return (self = lowerBound) (self = upperBound) +@since(1.5) +def In(self, *values): + +A boolean expression that is evaluated to true if the value of this +expression is any of the given columns. +NOTE: Normally, we shold name this function the small case `in`. However, `in` is +a reserved word in Python. So we can't help naming this the upper case `In`. + + df.select(df.name, df.age, df.age.In(2, 4)).show() ++-+---+-+ +| name|age|(age = 2)| ++-+---+-+ +|Alice| 2| true| +| Bob| 5|false| ++-+---+-+ + +for v in values: --- End diff -- I will try. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6908#issuecomment-113651585 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user yu-iskw closed the pull request at: https://github.com/apache/spark/pull/6908 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/6908#issuecomment-113643004 We have already implemented `inSet` which is exactly the same as `in`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
GitHub user yu-iskw opened a pull request: https://github.com/apache/spark/pull/6908 [SPARK-8477][sql][pyspark] Add in operator to DataFrame Column in Python You can merge this pull request into a Git repository by running: $ git pull https://github.com/yu-iskw/spark SPARK-8477 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6908.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6908 commit 95534b2dc0023a49100214ef5ea82433c2b5dc3d Author: Yu ISHIKAWA yuu.ishik...@gmail.com Date: 2015-06-19T17:47:01Z [SPARK-8477][sql][pyspark] Add in operator to DataFrame Column in Python commit 4d5cdc7910c08603862d1a1ba7c7d24a06bfa3bf Author: Yu ISHIKAWA yuu.ishik...@gmail.com Date: 2015-06-19T19:46:04Z Rename the method name to `In` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6908#issuecomment-113626549 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8477][sql][pyspark] Add in operator to ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6908#issuecomment-113626564 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
