[GitHub] spark issue #18128: [SPARK-20906][SparkR]:Constrained Logistic Regression fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18128 **[Test build #77495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77495/testReport)** for PR 18128 at commit [`8094b8d`](https://github.com/apache/spark/commit/8094b8d100bbb4ce1201ccbcf9c593c159b2e60a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18128: [SPARK-20906][SparkR]:Constrained Logistic Regression fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18128 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18128: [SPARK-20906][SparkR]:Constrained Logistic Regression fo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77495/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18047 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77494/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18047 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18047 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18047 **[Test build #77496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77496/testReport)** for PR 18047 at commit [`6d0b012`](https://github.com/apache/spark/commit/6d0b0125e1e50e6613524c174c6da637f43b1b68). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18118 cc @sethah also --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18118 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77497/testReport)** for PR 18118 at commit [`b0444fa`](https://github.com/apache/spark/commit/b0444fa75f4cc33a0c35cf88664a89a1c425e7a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77497/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77497/testReport)** for PR 18118 at commit [`b0444fa`](https://github.com/apache/spark/commit/b0444fa75f4cc33a0c35cf88664a89a1c425e7a1). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18074: [DOCS][MINOR] Scaladoc fixes (aka typo hunting)
Github user jaceklaskowski commented on the issue: https://github.com/apache/spark/pull/18074 Hey @srowen could you review the changes again and accept possibly? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user phatak-dev commented on the issue: https://github.com/apache/spark/pull/17972 @MLnick i am working on fixes you recommended. Meantime jenkins build doesn't seems to be started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/17972 jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17972 **[Test build #77498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77498/testReport)** for PR 17972 at commit [`4e48f5c`](https://github.com/apache/spark/commit/4e48f5c42d746ac0ab48c30e87d781e47bab4820). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77499/testReport)** for PR 17308 at commit [`a10276a`](https://github.com/apache/spark/commit/a10276abc783ad3818821c31f38ce56acc3e9d77). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/17308 Thanks @viirya and @zsxwing. I have tried to address you comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17972 **[Test build #77498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77498/testReport)** for PR 17972 at commit [`4e48f5c`](https://github.com/apache/spark/commit/4e48f5c42d746ac0ab48c30e87d781e47bab4820). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17972 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17972 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77498/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77499/testReport)** for PR 17308 at commit [`a10276a`](https://github.com/apache/spark/commit/a10276abc783ad3818821c31f38ce56acc3e9d77). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17308 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77499/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17972: [SPARK-20723][ML]Add intermediate storage level to tree ...
Github user phatak-dev commented on the issue:
https://github.com/apache/spark/pull/17972
There are around 33 binary incompatibility issues as I made this a shared
param. Some of them from build logs are below
```scala
[error] * abstract synthetic method
org$apache$spark$ml$param$shared$HasIntermediateStorageLevel$_setter_$intermediateStorageLevel_=(org.apache.spark.ml.param.Param)Unit
in trait org.apache.spark.ml.param.shared.HasIntermediateStorageLevel is
inherited by class ALSParams in current version.
[error]filter with:
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.param.shared.HasIntermediateStorageLevel.org$apache$spark$ml$param$shared$HasIntermediateStorageLevel$_setter_$intermediateStorageLevel_=")
[error] * method getIntermediateStorageLevel()java.lang.String in class
org.apache.spark.ml.recommendation.ALS is declared final in current version
[error]filter with:
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.recommendation.ALS.getIntermediateStorageLevel")
[error] * method intermediateStorageLevel()org.apache.spark.ml.param.Param
in class org.apache.spark.ml.recommendation.ALS is declared final in current
version
[error]filter with:
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.recommendation.ALS.intermediateStorageLevel")
[error] * abstract synthetic method
org$apache$spark$ml$param$shared$HasIntermediateStorageLevel$_setter_$intermediateStorageLevel_=(org.apache.spark.ml.param.Param)Unit
in trait org.apache.spark.ml.param.shared.HasIntermediateStorageLevel is
inherited by class RandomForestRegressorParams in current version.
```
Should I add them to the exclude file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18074: [DOCS][MINOR] Scaladoc fixes (aka typo hunting)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18074 **[Test build #77500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77500/testReport)** for PR 18074 at commit [`379e775`](https://github.com/apache/spark/commit/379e775ca8fad55f59b5d9f765ef662b866f3224). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77501/testReport)** for PR 18118 at commit [`9b3d03b`](https://github.com/apache/spark/commit/9b3d03ba99e69176e94b518a3f8cd2ac9e55d10e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77501/testReport)** for PR 18118 at commit [`9b3d03b`](https://github.com/apache/spark/commit/9b3d03ba99e69176e94b518a3f8cd2ac9e55d10e). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77501/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18047 **[Test build #77496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77496/testReport)** for PR 18047 at commit [`6d0b012`](https://github.com/apache/spark/commit/6d0b0125e1e50e6613524c174c6da637f43b1b68). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18047 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77496/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18047 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17993: [SPARK-20758][SQL] Add Constant propagation optimization
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/17993 LGTM - merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17100 **[Test build #77502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77502/testReport)** for PR 17100 at commit [`0cb9825`](https://github.com/apache/spark/commit/0cb9825a1cfebe67f08ce344e67a06c53e4a78be). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17993: [SPARK-20758][SQL] Add Constant propagation optim...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17993 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77503/testReport)** for PR 18118 at commit [`426bc68`](https://github.com/apache/spark/commit/426bc68e4b75fba76d993e0da94d28982b449c72). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77503/testReport)** for PR 18118 at commit [`426bc68`](https://github.com/apache/spark/commit/426bc68e4b75fba76d993e0da94d28982b449c72). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77503/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...
Github user setjet commented on a diff in the pull request:
https://github.com/apache/spark/pull/18113#discussion_r118914424
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
---
@@ -99,3 +97,67 @@ class TypedAverage[IN](val f: IN => Double) extends
Aggregator[IN, (Double, Long
toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
}
}
+
+class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN,
Double, Double] {
+ override def zero: Double = Double.PositiveInfinity
--- End diff --
Turns out I made a typo which caused me to miss a permutation of handling
null in the parameters...
Comparing both solutions (tuple with `OUT` as `java.lang.Double` vs
non-tuple with both `BUF` and `OUT` as `java.lang.Double`), it seems we have
the following trade-offs:
- tuple will require more data to be shuffled around as we are adding an
additional value
- non-tuple solution requires the developer to know a bit about the
internals, i.e.:
`val tuple = (x: (Double, Double)) => x._2
emptyDataSet.agg(typed.min(tuple)).show()`
`val nontuple = (x: (Double, java.lang.Double)) => x._2
emptyDataSet.agg(typed.min(nontuple)).show()`
This is because function `f` passed in into typed.min outputs a `BUF`,
forcing the caller to know about it the internals.
Given that users can always implement their own (non-tuple version) if
needed, I'd argue in favor of the tupled solution beacuse it is a bit more
developer friendly. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18074: [DOCS][MINOR] Scaladoc fixes (aka typo hunting)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18074 **[Test build #77500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77500/testReport)** for PR 18074 at commit [`379e775`](https://github.com/apache/spark/commit/379e775ca8fad55f59b5d9f765ef662b866f3224). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18074: [DOCS][MINOR] Scaladoc fixes (aka typo hunting)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18074 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77500/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18074: [DOCS][MINOR] Scaladoc fixes (aka typo hunting)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18074 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77504/testReport)** for PR 18118 at commit [`16ccbdf`](https://github.com/apache/spark/commit/16ccbdfd8862c528c90fdde94c8ec20d6631126e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17100 **[Test build #77502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77502/testReport)** for PR 17100 at commit [`0cb9825`](https://github.com/apache/spark/commit/0cb9825a1cfebe67f08ce344e67a06c53e4a78be). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77502/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17100 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18113#discussion_r118931501
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
---
@@ -99,3 +97,67 @@ class TypedAverage[IN](val f: IN => Double) extends
Aggregator[IN, (Double, Long
toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
}
}
+
+class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN,
Double, Double] {
+ override def zero: Double = Double.PositiveInfinity
--- End diff --
I don't get it. Why will using `java.lang.Double` as `BUT` and `OUT` leak
internals?
The signature of your `f` looks weird. Why is it `(Double,
java.lang.Double) => java.lang.Double`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18113#discussion_r118932092
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
---
@@ -99,3 +97,67 @@ class TypedAverage[IN](val f: IN => Double) extends
Aggregator[IN, (Double, Long
toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
}
}
+
+class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN,
Double, Double] {
+ override def zero: Double = Double.PositiveInfinity
--- End diff --
A modified version of existing `TypedSumDouble` to use `java.lang.Double`
as `BUF` and `OUT` looks like below. It returns `null` when applying on an
empty dataset.
class TypedSumDouble[IN](val f: IN => Double)
extends Aggregator[IN, java.lang.Double, java.lang.Double] {
override def zero: java.lang.Double = null
override def reduce(b: java.lang.Double, a: IN): java.lang.Double = {
if (b == null) {
f(a)
} else {
b + f(a)
}
}
override def merge(b1: java.lang.Double, b2: java.lang.Double):
java.lang.Double = {
if (b1 == null) {
b2
} else if (b2 == null) {
b1
} else {
b1 + b2
}
}
override def finish(reduction: java.lang.Double): java.lang.Double = {
reduction
}
override def bufferEncoder: Encoder[java.lang.Double] =
ExpressionEncoder[java.lang.Double]()
override def outputEncoder: Encoder[java.lang.Double] =
ExpressionEncoder[java.lang.Double]()
// Java api support
def this(f: MapFunction[IN, java.lang.Double]) = this(x => f.call(x))
def toColumnJava: TypedColumn[IN, java.lang.Double] = {
toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18080: [Spark-20771][SQL] Make weekofyear more intuitive
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18080 Is this variant available in any other DB? A lot of the goal of providing built-in functions is compatibility. Beyond that a lot of things are better handled with UDFs for special cases, not new built-ins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18086: [SPARK-20854][SQL] Extend hint syntax to support ...
Github user bogdanrdc commented on a diff in the pull request:
https://github.com/apache/spark/pull/18086#discussion_r118939619
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala
---
@@ -25,7 +25,7 @@ import org.apache.spark.sql.internal.SQLConf
* should be removed This node will be eliminated post analysis.
* A pair of (name, parameters).
*/
-case class UnresolvedHint(name: String, parameters: Seq[String], child:
LogicalPlan)
+case class UnresolvedHint(name: String, parameters: Seq[Any], child:
LogicalPlan)
--- End diff --
One useful hint parameter is a list of columns.
Something like `df.hint("hint", $"table", Seq($"col1", $"col2", $"col3"))`
In this case UnresolvedHint could be called like this:
```UnresolvedHint(name = String, parameters = Seq(Expression,
Seq[Expression]), child)```
But if `UnresolvedHint.parameters` is `Seq[Expression]` then it's not
possible to have this kind of hint.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18113: [SPARK-20890][SQL] Added min and max typed aggreg...
Github user setjet commented on a diff in the pull request:
https://github.com/apache/spark/pull/18113#discussion_r118939565
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
---
@@ -99,3 +97,67 @@ class TypedAverage[IN](val f: IN => Double) extends
Aggregator[IN, (Double, Long
toColumn.asInstanceOf[TypedColumn[IN, java.lang.Double]]
}
}
+
+class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN,
Double, Double] {
+ override def zero: Double = Double.PositiveInfinity
--- End diff --
Ah I see my misunderstanding: in reduce I tried to also have an `if` for
`f(a) == null` because of the previously mentioned implicit casting issue. This
would force a `java.lang.Double` to be returned by the function, as `Double ==
null` doesn't make sense in Scala.
I have updated the code, please have a look :) Becuase `OUT` is already a
`java.lang.Double`, we do not need the `toColumnJava`. As a result of `OUT`
being `java.lang.Double` however, we do need a `toColumnScala` to accommodate `
val f = (x: (Double, Double)) => x._2; empty.agg(typed.min(f)).show()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18118 **[Test build #77504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77504/testReport)** for PR 18118 at commit [`16ccbdf`](https://github.com/apache/spark/commit/16ccbdfd8862c528c90fdde94c8ec20d6631126e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: SPARK-20199 : Provided featureSubsetStrategy to GBTClass...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18118 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77504/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18080: [Spark-20771][SQL] Make weekofyear more intuitive
Github user setjet commented on the issue: https://github.com/apache/spark/pull/18080 This variant is available in other DB's, albeit with slightly different function and parameter naming. For example, MySQL allows it via the `week()` function: http://www.w3resource.com/mysql/date-and-time-functions/mysql-week-function.php In this case, you pass in an integer that specifies which permutation you want. Please note that if you look at the table, the 'Week 1 is the first week â¦' column is the difference between gregorian and iso. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...
Github user arzt commented on the issue: https://github.com/apache/spark/pull/17774 It's been a while. What can I do to draw some attention to this request? Is this issue not relevant enough? Thanks for reconsideration @felixcheung @brkyvz @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18137: [SPARK-20787][PYTHON] PySpark can't handle dateti...
GitHub user rberenguel opened a pull request: https://github.com/apache/spark/pull/18137 [SPARK-20787][PYTHON] PySpark can't handle datetimes before 1900 `time.mktime` can't handle dates from 1899-100, according to the documentation by design. `calendar.timegm` is equivalent in shared cases, but can handle those years. ## What changes were proposed in this pull request? Change `time.mktime` for the more able `calendar.timegm` to adress cases like: ```python import datetime as dt sqlContext.createDataFrame(sc.parallelize([[dt.datetime(1899,12,31)]])).count() ``` failing due to internal conversion failure when there is no timezone information in the time object. In the case there is information, `calendar` was used instead. ## How was this patch tested? The existing test cases should cover this change, since it should not change any existing functionality. This PR is original work from me and I license this work to the Spark project You can merge this pull request into a Git repository by running: $ git pull https://github.com/rberenguel/spark SPARK-20787-invalid-years Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18137 commit 6c0312f94e3fce2bf4d6a30055bd747be535bb0f Author: Ruben Berenguel Montoro Date: 2017-05-29T15:46:21Z SPARK-20787 time.mktime canât handle dates from 1899-100, by construction. calendar.timegm is equivalent in shared cases, but can handle those --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18137: [SPARK-20787][PYTHON] PySpark can't handle datetimes bef...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18137 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18138: [SPARK-20915][SQL] Make lpad/rpad with empty pad ...
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/18138
[SPARK-20915][SQL] Make lpad/rpad with empty pad string same as MySQL.
## What changes were proposed in this pull request?
Spark SQL `rpad/lpad` with empty pad string:
```sql
spark-sql> select rpad('hello', -2, ''), rpad('hello', -1, '') ,
rpad('hello', 0, ''), rpad('hello', 1, ''), rpad('hello', 3, ''), rpad('hello',
5, ''), rpad('hello', 6, '');
h hel hello hello
spark-sql> select lpad('hello', -2, ''), lpad('hello', -1, '') ,
lpad('hello', 0, ''), lpad('hello', 1, ''), lpad('hello', 3, ''), lpad('hello',
5, ''), lpad('hello', 6, '');
h hel hello hello
spark-sql>
```
but the MySQL result:
```
mysql> select rpad('hello', -2, ''), rpad('hello', -1, '') , rpad('hello',
0, ''), rpad('hello', 1, ''), rpad('hello', 3, ''), rpad('hello', 5, ''),
rpad('hello', 6, '') from dual;
+---+---+--+--+--+--+--+
| rpad('hello', -2, '') | rpad('hello', -1, '') | rpad('hello', 0, '') |
rpad('hello', 1, '') | rpad('hello', 3, '') | rpad('hello', 5, '') |
rpad('hello', 6, '') |
+---+---+--+--+--+--+--+
| NULL | NULL | | h
| hel | hello| NULL
|
+---+---+--+--+--+--+--+
1 row in set (0.00 sec)
mysql> select lpad('hello', -2, ''), lpad('hello', -1, '') , lpad('hello',
0, ''), lpad('hello', 1, ''), lpad('hello', 3, ''), lpad('hello', 5, ''),
lpad('hello', 6, '') from dual;
+---+---+--+--+--+--+--+
| lpad('hello', -2, '') | lpad('hello', -1, '') | lpad('hello', 0, '') |
lpad('hello', 1, '') | lpad('hello', 3, '') | lpad('hello', 5, '') |
lpad('hello', 6, '') |
+---+---+--+--+--+--+--+
| NULL | NULL | | h
| hel | hello| NULL
|
+---+---+--+--+--+--+--+
1 row in set (0.01 sec)
```
This PR fix this issue, after this PR:
```sql
spark-sql> select rpad('hello', -2, ''), rpad('hello', -1, '') ,
rpad('hello', 0, ''), rpad('hello', 1, ''), rpad('hello', 3, ''), rpad('hello',
5, ''), rpad('hello', 6, '');
NULLNULLh hel hello NULL
spark-sql> select lpad('hello', -2, ''), lpad('hello', -1, '') ,
lpad('hello', 0, ''), lpad('hello', 1, ''), lpad('hello', 3, ''), lpad('hello',
5, ''), lpad('hello', 6, '');
NULLNULLh hel hello NULL
spark-sql>
```
## How was this patch tested?
unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-20915
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18138.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18138
commit 3ac9fb07ef2f53315247ad12d391b1bed92319e9
Author: Yuming Wang
Date: 2017-05-29T16:47:55Z
Make lpad/rpad with empty pad string same as MySQL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18138: [SPARK-20915][SQL] Make lpad/rpad with empty pad string ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18138 **[Test build #77505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77505/testReport)** for PR 18138 at commit [`3ac9fb0`](https://github.com/apache/spark/commit/3ac9fb07ef2f53315247ad12d391b1bed92319e9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18122 **[Test build #77506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77506/testReport)** for PR 18122 at commit [`3510e24`](https://github.com/apache/spark/commit/3510e24379a26551edd7abf2bf8f3fb08ec42aba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18122 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77506/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18122 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18122 **[Test build #77506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77506/testReport)** for PR 18122 at commit [`3510e24`](https://github.com/apache/spark/commit/3510e24379a26551edd7abf2bf8f3fb08ec42aba). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18137: [SPARK-20787][PYTHON] PySpark can't handle dateti...
Github user rberenguel closed the pull request at: https://github.com/apache/spark/pull/18137 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18137: [SPARK-20787][PYTHON] PySpark can't handle datetimes bef...
Github user rberenguel commented on the issue: https://github.com/apache/spark/pull/18137 Closing while I fight with an issue seemingly related to DST between gmtime and mktime --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18047: [SPARK-20750][SQL] Built-in SQL Function Support - REPLA...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18047 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17953: [SPARK-20680][SQL] Spark-sql do not support for void col...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17953 ping @LantaoJin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18047: [SPARK-20750][SQL] Built-in SQL Function Support ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18047 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17880: [SPARK-20620][TEST]Improve some unit tests for NullExpre...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17880 LGTM pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17880: [SPARK-20620][TEST]Improve some unit tests for NullExpre...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17880 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17880: [SPARK-20620][TEST]Improve some unit tests for NullExpre...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17880 **[Test build #77507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77507/testReport)** for PR 17880 at commit [`3110f0f`](https://github.com/apache/spark/commit/3110f0f0c1a09b28a5706674ae65fd47ce48b163). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13706: [SPARK-15988] [SQL] Implement DDL commands: Create/Drop ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/13706 Sure. Will do it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18138: [SPARK-20915][SQL] Make lpad/rpad with empty pad string ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18138 **[Test build #77505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77505/testReport)** for PR 18138 at commit [`3ac9fb0`](https://github.com/apache/spark/commit/3ac9fb07ef2f53315247ad12d391b1bed92319e9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18138: [SPARK-20915][SQL] Make lpad/rpad with empty pad string ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18138 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18138: [SPARK-20915][SQL] Make lpad/rpad with empty pad string ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18138 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77505/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18135: [SPARK-20907][test] Use testQuietly for test suites that...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18135 LGTM. Merging to master and 2.2. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18135: [SPARK-20907][test] Use testQuietly for test suit...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18135 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18139: Spark 20787 invalid years
GitHub user rberenguel opened a pull request: https://github.com/apache/spark/pull/18139 Spark 20787 invalid years `time.mktime` can't handle dates from 1899-100, according to the documentation by design. `calendar.timegm` is equivalent in shared cases, but can handle those years. ## What changes were proposed in this pull request? Change `time.mktime` for the more able `calendar.timegm` to adress cases like: ```python import datetime as dt sqlContext.createDataFrame(sc.parallelize([[dt.datetime(1899,12,31)]])).count() ``` failing due to internal conversion failure when there is no timezone information in the time object. In the case there is information, `calendar` was used instead. ## How was this patch tested? The existing test cases cover this change, since it does not change any existing functionality. Added a test to confirm it working in the problematic range. This PR is original work from me and I license this work to the Spark project You can merge this pull request into a Git repository by running: $ git pull https://github.com/rberenguel/spark SPARK-20787-invalid-years Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18139.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18139 commit 6c0312f94e3fce2bf4d6a30055bd747be535bb0f Author: Ruben Berenguel Montoro Date: 2017-05-29T15:46:21Z SPARK-20787 time.mktime canât handle dates from 1899-100, by construction. calendar.timegm is equivalent in shared cases, but can handle those commit d3c41b5f18971168870524ad3a5fac876859bf4b Author: Ruben Berenguel Montoro Date: 2017-05-29T19:42:54Z SPARK-20787 Technically a hack. Using gmtime everywhere does not work well with DST shifts. So, for timeranges that donât work well with mktime, use gmtime --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18139: Spark 20787 invalid years
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18139 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18122 **[Test build #77508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77508/testReport)** for PR 18122 at commit [`320203e`](https://github.com/apache/spark/commit/320203eeea6d7613bb091f01b170fbfa2805b2a0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18122 **[Test build #77508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77508/testReport)** for PR 18122 at commit [`320203e`](https://github.com/apache/spark/commit/320203eeea6d7613bb091f01b170fbfa2805b2a0). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkMLTests(ReusedPySparkTestCase):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18122 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77508/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18122 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17880: [SPARK-20620][TEST]Improve some unit tests for NullExpre...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17880 **[Test build #77507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77507/testReport)** for PR 17880 at commit [`3110f0f`](https://github.com/apache/spark/commit/3110f0f0c1a09b28a5706674ae65fd47ce48b163). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17880: [SPARK-20620][TEST]Improve some unit tests for NullExpre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17880 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17880: [SPARK-20620][TEST]Improve some unit tests for NullExpre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17880 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77507/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13706: [SPARK-15988] [SQL] Implement DDL commands: Creat...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/13706#discussion_r118987906 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala --- @@ -52,3 +52,6 @@ class NoSuchPartitionsException(db: String, table: String, specs: Seq[TableParti class NoSuchTempFunctionException(func: String) extends AnalysisException(s"Temporary function '$func' not found") + +class NoSuchTempMacroException(func: String) --- End diff -- Please remove it. For reasons, please see the PR https://github.com/apache/spark/pull/17716. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13706: [SPARK-15988] [SQL] Implement DDL commands: Creat...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13706#discussion_r118989143
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -1090,6 +1090,24 @@ class SessionCatalog(
}
}
+ /** Create a temporary macro. */
+ def createTempMacro(
+ name: String,
+ info: ExpressionInfo,
+ functionBuilder: FunctionBuilder): Unit = {
+if (functionRegistry.functionExists(name)) {
--- End diff --
```
hive> create temporary macro max(x int)
> x*x;
OK
Time taken: 0.014 seconds
hive> select max(3) from t1;
OK
9
Time taken: 0.468 seconds, Fetched: 1 row(s)
hive> select max(3,4) from t1;
FAILED: SemanticException [Error 10015]: Line 1:7 Arguments length mismatch
'4': The macro max accepts exactly 1 arguments.
```
Hive overwrites the temporary function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13706: [SPARK-15988] [SQL] Implement DDL commands: Creat...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13706#discussion_r118989451
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -1090,6 +1090,24 @@ class SessionCatalog(
}
}
+ /** Create a temporary macro. */
+ def createTempMacro(
+ name: String,
+ info: ExpressionInfo,
+ functionBuilder: FunctionBuilder): Unit = {
+if (functionRegistry.functionExists(name)) {
+ throw new AnalysisException(s"Function $name already exists")
+}
+functionRegistry.registerFunction(name, info, functionBuilder)
+ }
+
+ /** Drop a temporary macro. */
+ def dropTempMacro(name: String, ignoreIfNotExists: Boolean): Unit = {
+if (!functionRegistry.dropMacro(name) && !ignoreIfNotExists) {
+ throw new NoSuchTempMacroException(name)
--- End diff --
```
hive> DROP TEMPORARY MACRO max;
OK
Time taken: 0.01 seconds
hive> select max(3) from t1;
OK
3
```
After we drop the macro, the existing function works well. That means, we
did not delete the original built-in functions. The built-in function will not
be dropped by ` DROP TEMPORARY MACRO`. After we drop the macro with the same
name, the original function `max` is using the original built-in function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18132: [SPARK-8184][SQL] Add additional function description fo...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/18132 Thanks - merging in master/branch-2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18132: [SPARK-8184][SQL] Add additional function descrip...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18132 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17935: [SPARK-20690][SQL] Subqueries in FROM should have alias ...
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/17935 I was trying to run a test case from another database which _does_ support unaliased subqueries in the `FROM` clause and hit a confusing parser error due to this patch's behavior change. While I agree that we shouldn't necessarily support this syntax, I think that the current error message that we're returning isn't very good so I've file https://issues.apache.org/jira/browse/SPARK-20916 to improve it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17308: [SPARK-19968][SS] Use a cached instance of `KafkaProduce...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17308 LGTM. Merging to master and 2.2. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18122: [SPARK-20899][PySpark] PySpark supports stringIndexerOrd...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18122 **[Test build #77509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77509/testReport)** for PR 18122 at commit [`4af4b35`](https://github.com/apache/spark/commit/4af4b3500de27acb0128763be755ea8078736d60). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17935: [SPARK-20690][SQL] Subqueries in FROM should have alias ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17935 @JoshRosen Thanks for filing this issue. I'll look into it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17308: [SPARK-19968][SS] Use a cached instance of `Kafka...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17308 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
