[GitHub] [spark] AmplabJenkins removed a comment on pull request #29851: [WIP][SPARK-22674][PYTHON] Removed namedtuple hack and made cloudpickle the default serializer
AmplabJenkins removed a comment on pull request #29851: URL: https://github.com/apache/spark/pull/29851#issuecomment-697314005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
HyukjinKwon commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697314379 Looks pretty good. I will push some changes into your branches by myself. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore
SparkQA removed a comment on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-697178580 **[Test build #129014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129014/testReport)** for PR 26935 at commit [`5c70db0`](https://github.com/apache/spark/commit/5c70db0394916251487cf9ff78ed726e3f04ebe6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29850: [SPARK-32974][ML] FeatureHasher transform optimization
AmplabJenkins removed a comment on pull request #29850: URL: https://github.com/apache/spark/pull/29850#issuecomment-697313771 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore
AmplabJenkins removed a comment on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-697315495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore
AmplabJenkins commented on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-697315495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29851: [WIP][SPARK-22674][PYTHON] Removed namedtuple hack and made cloudpickle the default serializer
SparkQA commented on pull request #29851: URL: https://github.com/apache/spark/pull/29851#issuecomment-697320500 **[Test build #129029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129029/testReport)** for PR 29851 at commit [`d837158`](https://github.com/apache/spark/commit/d8371587abe4361307d9a8364b2ca84a3a636b39). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29851: [WIP][SPARK-22674][PYTHON] Removed namedtuple hack and made cloudpickle the default serializer
AmplabJenkins commented on pull request #29851: URL: https://github.com/apache/spark/pull/29851#issuecomment-697320718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29851: [WIP][SPARK-22674][PYTHON] Removed namedtuple hack and made cloudpickle the default serializer
SparkQA removed a comment on pull request #29851: URL: https://github.com/apache/spark/pull/29851#issuecomment-697313456 **[Test build #129029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129029/testReport)** for PR 29851 at commit [`d837158`](https://github.com/apache/spark/commit/d8371587abe4361307d9a8364b2ca84a3a636b39). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29851: [WIP][SPARK-22674][PYTHON] Removed namedtuple hack and made cloudpickle the default serializer
AmplabJenkins removed a comment on pull request #29851: URL: https://github.com/apache/spark/pull/29851#issuecomment-697320718 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29851: [WIP][SPARK-22674][PYTHON] Removed namedtuple hack and made cloudpickle the default serializer
AmplabJenkins removed a comment on pull request #29851: URL: https://github.com/apache/spark/pull/29851#issuecomment-697320723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129029/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29848: [MINOR][SQL][3.0] Improve examples for `percentile_approx()`
SparkQA commented on pull request #29848: URL: https://github.com/apache/spark/pull/29848#issuecomment-697321959 **[Test build #129016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129016/testReport)** for PR 29848 at commit [`b063ca1`](https://github.com/apache/spark/commit/b063ca11ee53e071ad846fe0153500a7a0f9fd04). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29848: [MINOR][SQL][3.0] Improve examples for `percentile_approx()`
SparkQA removed a comment on pull request #29848: URL: https://github.com/apache/spark/pull/29848#issuecomment-697184829 **[Test build #129016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129016/testReport)** for PR 29848 at commit [`b063ca1`](https://github.com/apache/spark/commit/b063ca11ee53e071ad846fe0153500a7a0f9fd04). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29848: [MINOR][SQL][3.0] Improve examples for `percentile_approx()`
AmplabJenkins removed a comment on pull request #29848: URL: https://github.com/apache/spark/pull/29848#issuecomment-697323375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29848: [MINOR][SQL][3.0] Improve examples for `percentile_approx()`
AmplabJenkins commented on pull request #29848: URL: https://github.com/apache/spark/pull/29848#issuecomment-697323375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29850: [SPARK-32974][ML] FeatureHasher transform optimization
SparkQA commented on pull request #29850: URL: https://github.com/apache/spark/pull/29850#issuecomment-697326628 **[Test build #129025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129025/testReport)** for PR 29850 at commit [`eb60818`](https://github.com/apache/spark/commit/eb6081892e90c796ea5326a7d83bd88b6f3d3a7f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29850: [SPARK-32974][ML] FeatureHasher transform optimization
SparkQA removed a comment on pull request #29850: URL: https://github.com/apache/spark/pull/29850#issuecomment-697292058 **[Test build #129025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129025/testReport)** for PR 29850 at commit [`eb60818`](https://github.com/apache/spark/commit/eb6081892e90c796ea5326a7d83bd88b6f3d3a7f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29850: [SPARK-32974][ML] FeatureHasher transform optimization
AmplabJenkins commented on pull request #29850: URL: https://github.com/apache/spark/pull/29850#issuecomment-697327197 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29850: [SPARK-32974][ML] FeatureHasher transform optimization
AmplabJenkins removed a comment on pull request #29850: URL: https://github.com/apache/spark/pull/29850#issuecomment-697327197 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
SparkQA commented on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697334484 **[Test build #129030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129030/testReport)** for PR 25840 at commit [`edd8aa5`](https://github.com/apache/spark/commit/edd8aa513f8856a9b26e185b2974c6cbc4fb53e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
AmplabJenkins commented on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697335325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
AmplabJenkins removed a comment on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697335325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API
SparkQA commented on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-697342961 **[Test build #129020 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129020/testReport)** for PR 29756 at commit [`fad1976`](https://github.com/apache/spark/commit/fad1976e467ca73a26fbebd33ff07fc9014ae0b7). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait V2TableWithV1Fallback extends Table ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API
SparkQA removed a comment on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-697201587 **[Test build #129020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129020/testReport)** for PR 29756 at commit [`fad1976`](https://github.com/apache/spark/commit/fad1976e467ca73a26fbebd33ff07fc9014ae0b7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API
AmplabJenkins commented on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-697344170 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API
AmplabJenkins removed a comment on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-697344170 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29852: [SPARK-21481][ML][FOLLOWUP][Trivial] HashingTF use util.collection.OpenHashMap instead of mutable.HashMap
SparkQA commented on pull request #29852: URL: https://github.com/apache/spark/pull/29852#issuecomment-697347990 **[Test build #129028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129028/testReport)** for PR 29852 at commit [`000ca35`](https://github.com/apache/spark/commit/000ca3537d49a32b30b4f4bead00a2bdff790a43). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29852: [SPARK-21481][ML][FOLLOWUP][Trivial] HashingTF use util.collection.OpenHashMap instead of mutable.HashMap
AmplabJenkins commented on pull request #29852: URL: https://github.com/apache/spark/pull/29852#issuecomment-697348502 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29852: [SPARK-21481][ML][FOLLOWUP][Trivial] HashingTF use util.collection.OpenHashMap instead of mutable.HashMap
SparkQA removed a comment on pull request #29852: URL: https://github.com/apache/spark/pull/29852#issuecomment-697307454 **[Test build #129028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129028/testReport)** for PR 29852 at commit [`000ca35`](https://github.com/apache/spark/commit/000ca3537d49a32b30b4f4bead00a2bdff790a43). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29852: [SPARK-21481][ML][FOLLOWUP][Trivial] HashingTF use util.collection.OpenHashMap instead of mutable.HashMap
AmplabJenkins removed a comment on pull request #29852: URL: https://github.com/apache/spark/pull/29852#issuecomment-697348502 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] steveloughran commented on a change in pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1
steveloughran commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r493564469
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##
@@ -308,7 +307,7 @@ private[spark] class ApplicationMaster(
// The client-mode AM doesn't listen for incoming connections, so report
an invalid port.
registerAM(Utils.localHostName, -1, sparkConf,
sparkConf.getOption("spark.driver.appUIAddress"), appAttemptId)
- addAmIpFilter(Some(driverRef),
ProxyUriUtils.getPath(appAttemptId.getApplicationId))
+ addAmIpFilter(Some(driverRef), s"/proxy/$appAttemptId")
Review comment:
This a regression in the Hadoop APIs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #29843: [WIP][SPARK-29250] Upgrade to Hadoop 3.2.1
steveloughran commented on pull request #29843: URL: https://github.com/apache/spark/pull/29843#issuecomment-697352852 +expect a 3.2.2 before end of year. If there are regressions, that would be the place to get them addressed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
AngersZh commented on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697360429 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
SparkQA commented on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697361751 **[Test build #129031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129031/testReport)** for PR 25290 at commit [`6db1d49`](https://github.com/apache/spark/commit/6db1d49bee01c44657f9ce7b0a0547b56ca38618). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
AmplabJenkins commented on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697362539 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
AmplabJenkins removed a comment on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697362539 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badcase
srowen commented on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-697376204 Jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badcase
SparkQA commented on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-697376701 **[Test build #129032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129032/testReport)** for PR 29824 at commit [`250e397`](https://github.com/apache/spark/commit/250e397e32ac6b55ea84d68cacd340c4cbc37870). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badcase
AmplabJenkins commented on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-697379263 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badca
AmplabJenkins removed a comment on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-697379263 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697390632 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins removed a comment on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697390632 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
SparkQA commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697399954 **[Test build #129033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129033/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29798: [SPARK-32931][SQL] Unevaluable Expressions are not Foldable
SparkQA commented on pull request #29798: URL: https://github.com/apache/spark/pull/29798#issuecomment-697467645 **[Test build #129034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129034/testReport)** for PR 29798 at commit [`7ecb1e6`](https://github.com/apache/spark/commit/7ecb1e6987f87b7b1e3164ef227cc6fb45243a23). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29798: [SPARK-32931][SQL] Unevaluable Expressions are not Foldable
AmplabJenkins commented on pull request #29798: URL: https://github.com/apache/spark/pull/29798#issuecomment-697470258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29798: [SPARK-32931][SQL] Unevaluable Expressions are not Foldable
AmplabJenkins removed a comment on pull request #29798: URL: https://github.com/apache/spark/pull/29798#issuecomment-697470258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
SparkQA commented on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697533242 **[Test build #129030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129030/testReport)** for PR 25840 at commit [`edd8aa5`](https://github.com/apache/spark/commit/edd8aa513f8856a9b26e185b2974c6cbc4fb53e9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
SparkQA removed a comment on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697334484 **[Test build #129030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129030/testReport)** for PR 25840 at commit [`edd8aa5`](https://github.com/apache/spark/commit/edd8aa513f8856a9b26e185b2974c6cbc4fb53e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors
dongjoon-hyun commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-697533646 The decomission failure is irrelevant to this one. ```suggestion KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning *** FAILED *** ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
AmplabJenkins commented on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697535821 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25840: [SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table
AmplabJenkins removed a comment on pull request #25840: URL: https://github.com/apache/spark/pull/25840#issuecomment-697535821 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors
dongjoon-hyun commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-697539175 cc @holdenk , @dbtsai , @viirya , @sunchao This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors
holdenk commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-697539116 I agree this PR isn't touching anything in the decommissioning logic. That being said, I'll spend some time today on the decommissioning integration tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors
dongjoon-hyun commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-697540293 Thanks, @holdenk . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29286: [WIP][SPARK-21708][BUILD] Migrate build to sbt 1.x
srowen commented on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-697545434 Jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29286: [WIP][SPARK-21708][BUILD] Migrate build to sbt 1.x
SparkQA commented on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-697549208 **[Test build #129035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129035/testReport)** for PR 29286 at commit [`bfaa525`](https://github.com/apache/spark/commit/bfaa525659dc7f7817873055f3224d885a419376). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29286: [WIP][SPARK-21708][BUILD] Migrate build to sbt 1.x
AmplabJenkins commented on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-697551065 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29286: [WIP][SPARK-21708][BUILD] Migrate build to sbt 1.x
AmplabJenkins removed a comment on pull request #29286: URL: https://github.com/apache/spark/pull/29286#issuecomment-697551065 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fhoering commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
fhoering commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697567806 OK. I re-pushed some minor changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
SparkQA commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697572196 **[Test build #129036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129036/testReport)** for PR 29806 at commit [`804a14f`](https://github.com/apache/spark/commit/804a14f9253a8785c0241b5fe0fbb3ade99b96d6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
AmplabJenkins commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697574410 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
HyukjinKwon commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697574364 Thanks @fhoering for proofreading :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
AmplabJenkins removed a comment on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697574410 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bogdanghit commented on a change in pull request #29834: [SPARK-32963][SQL] empty string should be consistent for schema name in SparkGetSchemasOperation
bogdanghit commented on a change in pull request #29834:
URL: https://github.com/apache/spark/pull/29834#discussion_r493706395
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala
##
@@ -77,7 +77,8 @@ private[hive] class SparkGetSchemasOperation(
val globalTempViewDb =
sqlContext.sessionState.catalog.globalTempViewManager.database
val databasePattern =
Pattern.compile(CLIServiceUtils.patternToRegex(schemaName))
- if (databasePattern.matcher(globalTempViewDb).matches()) {
+ if (schemaName == null || schemaName.isEmpty ||
Review comment:
Won't this break upstream client applications that depend on this
behavior?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors
HyukjinKwon commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-697592931 Thanks @holdenk and @dongjoon-hyun. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
SparkQA removed a comment on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697572196 **[Test build #129036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129036/testReport)** for PR 29806 at commit [`804a14f`](https://github.com/apache/spark/commit/804a14f9253a8785c0241b5fe0fbb3ade99b96d6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
SparkQA commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697620145 **[Test build #129036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129036/testReport)** for PR 29806 at commit [`804a14f`](https://github.com/apache/spark/commit/804a14f9253a8785c0241b5fe0fbb3ade99b96d6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
AmplabJenkins commented on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697621801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging
AmplabJenkins removed a comment on pull request #29806: URL: https://github.com/apache/spark/pull/29806#issuecomment-697621801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
viirya commented on a change in pull request #29828:
URL: https://github.com/apache/spark/pull/29828#discussion_r493724659
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprs.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Simplify redundant json related expressions.
+ */
+object OptimizeJsonExprs extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case p => p.transformExpressions {
+ case JsonToStructs(_, options1, StructsToJson(options2, child,
timeZoneId2), timeZoneId1)
Review comment:
Ah, good point! I think we can only get rid of the chain if the schema
is the same.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29792: [SPARK-32858][SQL] UnwrapCastInBinaryComparison: support other numeric types
sunchao commented on a change in pull request #29792: URL: https://github.com/apache/spark/pull/29792#discussion_r493726234 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparisonSuite.scala ## @@ -67,6 +77,23 @@ class UnwrapCastInBinaryComparisonSuite extends PlanTest with ExpressionEvalHelp assertEquivalent(castInt(f) <=> v.toInt, f <=> v) assertEquivalent(castInt(f) <= v.toInt, f === v) assertEquivalent(castInt(f) < v.toInt, falseIfNotNull(f)) + +val d = Float.NegativeInfinity +assertEquivalent(castDouble(f2) > d.toDouble, f2 =!= d) Review comment: it is rounding down, see below for a test on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
viirya commented on a change in pull request #29828:
URL: https://github.com/apache/spark/pull/29828#discussion_r493728083
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprs.scala
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Simplify redundant json related expressions.
+ */
+object OptimizeJsonExprs extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case p => p.transformExpressions {
+ case JsonToStructs(_, options1, StructsToJson(options2, child,
timeZoneId2), timeZoneId1)
+ if options1 == options2 && timeZoneId1 == timeZoneId2 =>
Review comment:
Hmm, so my initial commit is limit to two options are both empty, do you
think it is safer? Per
https://github.com/apache/spark/pull/29828#discussion_r492673131, we can look
at the json options later and see whether the same option is symmetrical in
read and write.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] RussellSpitzer opened a new pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
RussellSpitzer opened a new pull request #29853: URL: https://github.com/apache/spark/pull/29853 The default is always ErrorsOnExist regardless of DataSource version. Fixing the JavaDoc to reflect this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
AmplabJenkins commented on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697645581 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] RussellSpitzer commented on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
RussellSpitzer commented on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697647382 @brkyvz - Could you please review this re: our conversation a while back :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] RussellSpitzer commented on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
RussellSpitzer commented on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697648811 @rdblue You may want to take a look as well if you have a moment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
rdblue commented on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697652647 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29792: [SPARK-32858][SQL] UnwrapCastInBinaryComparison: support other numeric types
cloud-fan commented on a change in pull request #29792:
URL: https://github.com/apache/spark/pull/29792#discussion_r493731636
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparisonSuite.scala
##
@@ -79,13 +106,65 @@ class UnwrapCastInBinaryComparisonSuite extends PlanTest
with ExpressionEvalHelp
assertEquivalent(castInt(f) < v, falseIfNotNull(f))
}
- test("unwrap casts when literal is within range (min, max)") {
-assertEquivalent(castInt(f) > 300, f > 300.toShort)
-assertEquivalent(castInt(f) >= 500, f >= 500.toShort)
-assertEquivalent(castInt(f) === 32766, f === 32766.toShort)
-assertEquivalent(castInt(f) <=> 32766, f <=> 32766.toShort)
-assertEquivalent(castInt(f) <= -6000, f <= -6000.toShort)
-assertEquivalent(castInt(f) < -32767, f < -32767.toShort)
+ test("unwrap casts when literal is within range (min, max) or fromType has
no range") {
+Seq(300, 500, 32766, -6000, -32767).foreach(v => {
+ assertEquivalent(castInt(f) > v, f > v.toShort)
+ assertEquivalent(castInt(f) >= v, f >= v.toShort)
+ assertEquivalent(castInt(f) === v, f === v.toShort)
+ assertEquivalent(castInt(f) <=> v, f <=> v.toShort)
+ assertEquivalent(castInt(f) <= v, f <= v.toShort)
+ assertEquivalent(castInt(f) < v, f < v.toShort)
+})
+
+Seq(3.14.toFloat.toDouble, -1000.0.toFloat.toDouble,
+ 20.0.toFloat.toDouble, -2.414.toFloat.toDouble,
+ Float.MinValue.toDouble, Float.MaxValue.toDouble,
Float.PositiveInfinity.toDouble
+).foreach(v => {
+ assertEquivalent(castDouble(f2) > v, f2 > v.toFloat)
+ assertEquivalent(castDouble(f2) >= v, f2 >= v.toFloat)
+ assertEquivalent(castDouble(f2) === v, f2 === v.toFloat)
+ assertEquivalent(castDouble(f2) <=> v, f2 <=> v.toFloat)
+ assertEquivalent(castDouble(f2) <= v, f2 <= v.toFloat)
+ assertEquivalent(castDouble(f2) < v, f2 < v.toFloat)
+})
+
+Seq(decimal2(100.20), decimal2(-200.50)).foreach(v => {
+ assertEquivalent(castDecimal2(f3) > v, f3 > decimal(v))
+ assertEquivalent(castDecimal2(f3) >= v, f3 >= decimal(v))
+ assertEquivalent(castDecimal2(f3) === v, f3 === decimal(v))
+ assertEquivalent(castDecimal2(f3) <=> v, f3 <=> decimal(v))
+ assertEquivalent(castDecimal2(f3) <= v, f3 <= decimal(v))
+ assertEquivalent(castDecimal2(f3) < v, f3 < decimal(v))
+})
+ }
+
+ test("unwrap cast when literal is within range (min, max) AND has round up
or down") {
+// Cases for rounding down
+var doubleValue = 100.6
+assertEquivalent(castDouble(f) > doubleValue, f > doubleValue.toShort)
+assertEquivalent(castDouble(f) > doubleValue, f > doubleValue.toShort)
+assertEquivalent(castDouble(f) === doubleValue, falseIfNotNull(f))
+assertEquivalent(castDouble(f) <=> doubleValue, false)
+assertEquivalent(castDouble(f) <= doubleValue, f <= doubleValue.toShort)
+assertEquivalent(castDouble(f) < doubleValue, f <= doubleValue.toShort)
+
+// Cases for rounding up: 3.14 will be rounded to 3.1410... after
casting to float
Review comment:
so casting double to float can be either rounding up or down, depend on
the value?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
SparkQA commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697653808 **[Test build #129033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129033/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
SparkQA removed a comment on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697399954 **[Test build #129033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129033/testReport)** for PR 29591 at commit [`fab00f1`](https://github.com/apache/spark/commit/fab00f1107beb301d093618d5c797799df768253). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
SparkQA commented on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697656590 **[Test build #129037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129037/testReport)** for PR 29853 at commit [`9bcb494`](https://github.com/apache/spark/commit/9bcb4949ed3417010df00b6a6ae4b0f4be1a8808). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697657470 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode
AmplabJenkins removed a comment on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697645581 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins removed a comment on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697657470 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
SparkQA commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697669398 **[Test build #129038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129038/testReport)** for PR 29828 at commit [`08cd0a7`](https://github.com/apache/spark/commit/08cd0a7172fd6fe3eb690a2fb27ccc3f14d536a8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
AmplabJenkins commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697671655 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
AmplabJenkins removed a comment on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697671655 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
AmplabJenkins commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697696466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
AmplabJenkins removed a comment on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697696466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0
dongjoon-hyun commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-697700651 According to @sarutak , this broke `org.apache.spark.deploy.history.ChromeUIHistoryServerSuite` due to Guava dependency. - https://github.com/apache/spark/pull/29827#discussion_r492654755 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain
SparkQA commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697707135 **[Test build #129039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129039/testReport)** for PR 29828 at commit [`078dc84`](https://github.com/apache/spark/commit/078dc84cd32d6fb0ccda20b0d161699a4e85355f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0
srowen commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-697711108 Shoot. If this is non-essential, I think we can just revert it. Or .. just lose this test? let me comment on the other PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #29827: [SPARK-32957][INFRA] Add a GitHub Actions job to run WebUI tests with Chrome
srowen commented on a change in pull request #29827:
URL: https://github.com/apache/spark/pull/29827#discussion_r493753499
##
File path: .github/workflows/build_and_test.yml
##
@@ -273,6 +273,44 @@ jobs:
cd docs
jekyll build
+ webui-tests-with-chrome:
+name: WebUI tests with chrome
+runs-on: ubuntu-latest
+steps:
+- name: Checkout Spark repository
+ uses: actions/checkout@v2
+- name: Cache Maven local repository
+ uses: actions/cache@v2
+ with:
+path: ~/.m2/repository
+key: webui-tests-with-chrome-maven-${{ hashFiles('**/pom.xml') }}
+restore-keys: |
+ webui-tests-with-chrome-maven-
+- name: Install Java 11
+ uses: actions/setup-java@v1
+ with:
+java-version: 11
+- name: Install Chrome and ChromeDriver
+ run: |
+sudo apt update
+sudo apt install google-chrome-stable
+sudo apt install chromium-chromedriver
+- name: Run WebUI tests with Maven
+ run: |
+export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g
-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+export MAVEN_CLI_OPTS="--no-transfer-progress"
+mkdir -p ~/.m2
+./build/mvn -Dspark.test.webdriver.chrome.driver=/usr/bin/chromedriver
\
+ -Dguava.version=25.0-jre -Djava.version=11
-Dtest.default.exclude.tags= -Dtest=none \
Review comment:
What about just not adding this test? :)
How important is it? does this cover new ground?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29792: [SPARK-32858][SQL] UnwrapCastInBinaryComparison: support other numeric types
sunchao commented on a change in pull request #29792:
URL: https://github.com/apache/spark/pull/29792#discussion_r493753795
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparisonSuite.scala
##
@@ -79,13 +106,65 @@ class UnwrapCastInBinaryComparisonSuite extends PlanTest
with ExpressionEvalHelp
assertEquivalent(castInt(f) < v, falseIfNotNull(f))
}
- test("unwrap casts when literal is within range (min, max)") {
-assertEquivalent(castInt(f) > 300, f > 300.toShort)
-assertEquivalent(castInt(f) >= 500, f >= 500.toShort)
-assertEquivalent(castInt(f) === 32766, f === 32766.toShort)
-assertEquivalent(castInt(f) <=> 32766, f <=> 32766.toShort)
-assertEquivalent(castInt(f) <= -6000, f <= -6000.toShort)
-assertEquivalent(castInt(f) < -32767, f < -32767.toShort)
+ test("unwrap casts when literal is within range (min, max) or fromType has
no range") {
+Seq(300, 500, 32766, -6000, -32767).foreach(v => {
+ assertEquivalent(castInt(f) > v, f > v.toShort)
+ assertEquivalent(castInt(f) >= v, f >= v.toShort)
+ assertEquivalent(castInt(f) === v, f === v.toShort)
+ assertEquivalent(castInt(f) <=> v, f <=> v.toShort)
+ assertEquivalent(castInt(f) <= v, f <= v.toShort)
+ assertEquivalent(castInt(f) < v, f < v.toShort)
+})
+
+Seq(3.14.toFloat.toDouble, -1000.0.toFloat.toDouble,
+ 20.0.toFloat.toDouble, -2.414.toFloat.toDouble,
+ Float.MinValue.toDouble, Float.MaxValue.toDouble,
Float.PositiveInfinity.toDouble
+).foreach(v => {
+ assertEquivalent(castDouble(f2) > v, f2 > v.toFloat)
+ assertEquivalent(castDouble(f2) >= v, f2 >= v.toFloat)
+ assertEquivalent(castDouble(f2) === v, f2 === v.toFloat)
+ assertEquivalent(castDouble(f2) <=> v, f2 <=> v.toFloat)
+ assertEquivalent(castDouble(f2) <= v, f2 <= v.toFloat)
+ assertEquivalent(castDouble(f2) < v, f2 < v.toFloat)
+})
+
+Seq(decimal2(100.20), decimal2(-200.50)).foreach(v => {
+ assertEquivalent(castDecimal2(f3) > v, f3 > decimal(v))
+ assertEquivalent(castDecimal2(f3) >= v, f3 >= decimal(v))
+ assertEquivalent(castDecimal2(f3) === v, f3 === decimal(v))
+ assertEquivalent(castDecimal2(f3) <=> v, f3 <=> decimal(v))
+ assertEquivalent(castDecimal2(f3) <= v, f3 <= decimal(v))
+ assertEquivalent(castDecimal2(f3) < v, f3 < decimal(v))
+})
+ }
+
+ test("unwrap cast when literal is within range (min, max) AND has round up
or down") {
+// Cases for rounding down
+var doubleValue = 100.6
+assertEquivalent(castDouble(f) > doubleValue, f > doubleValue.toShort)
+assertEquivalent(castDouble(f) > doubleValue, f > doubleValue.toShort)
+assertEquivalent(castDouble(f) === doubleValue, falseIfNotNull(f))
+assertEquivalent(castDouble(f) <=> doubleValue, false)
+assertEquivalent(castDouble(f) <= doubleValue, f <= doubleValue.toShort)
+assertEquivalent(castDouble(f) < doubleValue, f <= doubleValue.toShort)
+
+// Cases for rounding up: 3.14 will be rounded to 3.1410... after
casting to float
Review comment:
@cloud-fan Sorry i was wrong in the above comment (somehow I was
thinking casting from double to short there).
Yes, it appears that casting from double to float can be either rounding up
or down, depending on value:
```scala
scala> val x = 0.3999
x: Double = 0.3999
scala> val y = x.toFloat
y: Float = 0.3998
scala > val x = 0.4999
y: Double = 0.4999
scala> val y = x.toFloat
y: Float = 0.5
```
Also the test here is incorrect. casting 3.14 of double to float doesn't
change anything. I'll update it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0
dongjoon-hyun commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-697728071 Thanks, @srowen ~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29827: [SPARK-32957][INFRA] Add a GitHub Actions job to run WebUI tests with Chrome
dongjoon-hyun commented on pull request #29827: URL: https://github.com/apache/spark/pull/29827#issuecomment-697730993 @sarutak , @HyukjinKwon , @srowen . If this is only test issue, I guess we can revisit this after we upgrade Apache Spark's Guava version successfully. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
SparkQA commented on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697736614 **[Test build #129031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129031/testReport)** for PR 25290 at commit [`6db1d49`](https://github.com/apache/spark/commit/6db1d49bee01c44657f9ce7b0a0547b56ca38618). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
SparkQA removed a comment on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697361751 **[Test build #129031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129031/testReport)** for PR 25290 at commit [`6db1d49`](https://github.com/apache/spark/commit/6db1d49bee01c44657f9ce7b0a0547b56ca38618). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
AmplabJenkins commented on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697739218 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25290: [SPARK-28551][SQL]Add a Checker Rule when CTAS SQL with LOCATION
AmplabJenkins removed a comment on pull request #25290: URL: https://github.com/apache/spark/pull/25290#issuecomment-697739218 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
