[GitHub] [spark] cloud-fan commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
cloud-fan commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611343954 the change LGTM, can you regenerate the benchmark numbers? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled
SparkQA commented on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled URL: https://github.com/apache/spark/pull/26901#issuecomment-611343953 **[Test build #121006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121006/testReport)** for PR 26901 at commit [`c7e969a`](https://github.com/apache/spark/commit/c7e969abe4c85bc3948ff9cc247ef3fc32044043). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-611343370 Thanks @cloud-fan That is the approach I was going to start with. In the wild/production I hit this issue with a join, but I forget the exact reason/context. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #28114: [SPARK-31330] Automatically label PRs based on the paths they touch
HeartSaVioR edited a comment on issue #28114: [SPARK-31330] Automatically label PRs based on the paths they touch URL: https://github.com/apache/spark/pull/28114#issuecomment-611342090 > I just think without any background there that if the datasource is for streaming, why we don't add streaming as part of package name? The datasource will run in "batch query", though the input data is from "streaming query". I might have to reiterate; please don't get me wrong. I don't object the feature, I said it's huge one step forward. I just wanted to point out that we require manual label for module in the PR title and it's kinda accurate (otherwise committer would fix it) so it seems redundant to do classification here unless we also do automate on PR title. (If we are confident about the classification then why not?) Yes that might require another implementation of bot hence I'm not strong about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #28114: [SPARK-31330] Automatically label PRs based on the paths they touch
HeartSaVioR commented on issue #28114: [SPARK-31330] Automatically label PRs based on the paths they touch URL: https://github.com/apache/spark/pull/28114#issuecomment-611342090 > I just think without any background there that if the datasource is for streaming, why we don't add streaming as part of package name? The datasource will run in "batch query", though the input data is from "streaming query". I might have to reiterate; please don't get me wrong. I don't object the feature, I said it's huge one step forward. I just want to point out that we require manual label for module in the PR title and it's kinda accurate (otherwise committer would fix it) so it seems redundant to do classification here unless we also do automate on PR title. Yes that might require another implementation of bot hence I'm not strong about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference
AmplabJenkins removed a comment on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference URL: https://github.com/apache/spark/pull/28121#issuecomment-611341881 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled
AmplabJenkins commented on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled URL: https://github.com/apache/spark/pull/26901#issuecomment-611341932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25698/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-611341928 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611341982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611341982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-611341928 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled
AmplabJenkins removed a comment on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled URL: https://github.com/apache/spark/pull/26901#issuecomment-611341923 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled
AmplabJenkins removed a comment on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled URL: https://github.com/apache/spark/pull/26901#issuecomment-611341932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25698/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-611341920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-611341920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611341971 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference
AmplabJenkins removed a comment on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference URL: https://github.com/apache/spark/pull/28121#issuecomment-611341885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference
AmplabJenkins commented on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference URL: https://github.com/apache/spark/pull/28121#issuecomment-611341881 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference
AmplabJenkins commented on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference URL: https://github.com/apache/spark/pull/28121#issuecomment-611341885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled
AmplabJenkins commented on issue #26901: [SPARK-29152][2.4][test-maven]Executor Plugin shutdown when dynamic allocation is enabled URL: https://github.com/apache/spark/pull/26901#issuecomment-611341923 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611341971 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-611341506 **[Test build #121004 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121004/testReport)** for PR 28104 at commit [`207cae8`](https://github.com/apache/spark/commit/207cae82d88599970da67735f62f334f8cf0762a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference
SparkQA commented on issue #28121: [SPARK-31348][SQL][DOCS] Document Join in SQL Reference URL: https://github.com/apache/spark/pull/28121#issuecomment-611341505 **[Test build #121003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121003/testReport)** for PR 28121 at commit [`ffe4e09`](https://github.com/apache/spark/commit/ffe4e0995fd249e2fc6a0334007da002c8fcf278). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
SparkQA commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611341499 **[Test build #121005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121005/testReport)** for PR 28060 at commit [`af42f50`](https://github.com/apache/spark/commit/af42f50c53aa9be5fa7540591fe2a6277357377c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
beliefer commented on a change in pull request #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#discussion_r405968827 ## File path: sql/core/src/test/resources/sql-tests/results/limit.sql.out ## @@ -7,8 +7,8 @@ SELECT * FROM testdata LIMIT 2 -- !query schema struct -- !query output -1 1 -2 2 +51 51 Review comment: I use repartition resolved the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611339919 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25694/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins removed a comment on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611339913 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
beliefer commented on a change in pull request #28060: [SPARK-31291][SQL][TEST]
Avoid load test data if test case not uses them
URL: https://github.com/apache/spark/pull/28060#discussion_r405967900
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -668,6 +690,7 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
try {
TimeZone.setDefault(originalTimeZone)
Locale.setDefault(originalLocale)
+ unloadTestData(spark)
Review comment:
OK.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611339919 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25694/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
AmplabJenkins commented on issue #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#issuecomment-611339913 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611339034 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611339041 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120998/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
cloud-fan commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-611338971 Can you create a custom RDD which generates random data and set the preferred location to one executor? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611339041 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120998/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611339034 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
SparkQA removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611329612 **[Test build #120998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120998/testReport)** for PR 28160 at commit [`df340b8`](https://github.com/apache/spark/commit/df340b8e6ffa0429a5193da8a384170423e2515e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
SparkQA commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611338725 **[Test build #120998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120998/testReport)** for PR 28160 at commit [`df340b8`](https://github.com/apache/spark/commit/df340b8e6ffa0429a5193da8a384170423e2515e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
cloud-fan commented on a change in pull request #28060:
[SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
URL: https://github.com/apache/spark/pull/28060#discussion_r405966276
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -668,6 +690,7 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
try {
TimeZone.setDefault(originalTimeZone)
Locale.setDefault(originalLocale)
+ unloadTestData(spark)
Review comment:
I'd prefer `createTestTables` and `removeTestTables`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611338094 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611338100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121000/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
SparkQA removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611333492 **[Test build #121000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121000/testReport)** for PR 28161 at commit [`53897da`](https://github.com/apache/spark/commit/53897da43e41e86333fdcd4bbdab3980e62187c9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them
cloud-fan commented on a change in pull request #28060: [SPARK-31291][SQL][TEST] Avoid load test data if test case not uses them URL: https://github.com/apache/spark/pull/28060#discussion_r405965961 ## File path: sql/core/src/test/resources/sql-tests/results/limit.sql.out ## @@ -7,8 +7,8 @@ SELECT * FROM testdata LIMIT 2 -- !query schema struct -- !query output -1 1 -2 2 +51 51 Review comment: can we add a sort before limit? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
SparkQA commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611338015 **[Test build #121000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121000/testReport)** for PR 28161 at commit [`53897da`](https://github.com/apache/spark/commit/53897da43e41e86333fdcd4bbdab3980e62187c9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611338100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121000/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611338094 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
prakharjain09 commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-611337698 @dongjoon-hyun @holdenk Thanks for the review. I have addressed all the review comments. From last few days, few UTs are failing and they didn't seem related to my changes. Can it be possible that my tests are causing the other parallely running UTs to fail? Or these are unrelated intermittent failures which should be ignored? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] prakharjain09 commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned
prakharjain09 commented on issue #27864: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned URL: https://github.com/apache/spark/pull/27864#issuecomment-611336898 > Thank you for updating, @prakharjain09 . This is a best effort approach, so it's important to be clear on the limitation. Could you elaborate more about this PR's limitation at PR description? Done. Added more info in description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on issue #28119: [SPARK-31359][SQL] Speed up timestamps rebasing
cloud-fan edited a comment on issue #28119: [SPARK-31359][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-611334888 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611335841 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611335846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25693/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611335841 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611335846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25693/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
SparkQA commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611335488 **[Test build #121002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121002/testReport)** for PR 27571 at commit [`2d1f8b1`](https://github.com/apache/spark/commit/2d1f8b1e1546301dee709e7d4ae38dec5deafac0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
huaxingao commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611335078 LGTM. Thanks for the follow-up @gengliangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28119: [SPARK-31359][SQL] Speed up timestamps rebasing
cloud-fan commented on issue #28119: [SPARK-31359][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119#issuecomment-611334888 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28119: [SPARK-31359][SQL] Speed up timestamps rebasing
cloud-fan closed pull request #28119: [SPARK-31359][SQL] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611333786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611333783 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25692/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611333779 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611333783 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25692/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
AmplabJenkins removed a comment on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611333779 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611333786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins removed a comment on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611333790 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25691/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
AmplabJenkins commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611333790 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25691/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
SparkQA commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611333492 **[Test build #121000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121000/testReport)** for PR 28161 at commit [`53897da`](https://github.com/apache/spark/commit/53897da43e41e86333fdcd4bbdab3980e62187c9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR
SparkQA commented on issue #27571: [SPARK-30819][SPARKR][ML] Add FMRegressor wrapper to SparkR URL: https://github.com/apache/spark/pull/27571#issuecomment-611333490 **[Test build #121001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121001/testReport)** for PR 27571 at commit [`cb7ec6a`](https://github.com/apache/spark/commit/cb7ec6a6d2d3798a0633ef9c850dc3980cbc9fcd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611332687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120999/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
SparkQA commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611332614 **[Test build #120999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120999/testReport)** for PR 28130 at commit [`537c594`](https://github.com/apache/spark/commit/537c594d27dc9b05719939d08af73f08d7e81e4d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611332683 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611332683 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611332687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120999/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
SparkQA removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611329611 **[Test build #120999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120999/testReport)** for PR 28130 at commit [`537c594`](https://github.com/apache/spark/commit/537c594d27dc9b05719939d08af73f08d7e81e4d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
gengliangwang commented on issue #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161#issuecomment-611332076 cc @srowen @huaxingao This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang opened a new pull request #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide
gengliangwang opened a new pull request #28161: [SPARK-31333][Followup][Doc] Link Join Hints doc in SQL perf tuning guide URL: https://github.com/apache/spark/pull/28161 ### What changes were proposed in this pull request? This is a follow-up of https://github.com/apache/spark/pull/28113. There is also a brief section about Join hints in SQL perf tuning guide: https://spark.apache.org/docs/latest/sql-performance-tuning.html . We should link the new Join hint doc in it. ### Why are the changes needed? So that users can read more examples. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Manually build the doc and check it:  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611329903 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611329913 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25690/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611329908 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611329915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25689/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611329913 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25690/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611329908 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
AmplabJenkins removed a comment on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611329915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25689/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
AmplabJenkins removed a comment on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611329903 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
SparkQA commented on issue #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF URL: https://github.com/apache/spark/pull/28160#issuecomment-611329612 **[Test build #120998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120998/testReport)** for PR 28160 at commit [`df340b8`](https://github.com/apache/spark/commit/df340b8e6ffa0429a5193da8a384170423e2515e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference
SparkQA commented on issue #28130: [SPARK-31355][SQL][DOCS] Document TABLESAMPLE in SQL Reference URL: https://github.com/apache/spark/pull/28130#issuecomment-611329611 **[Test build #120999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120999/testReport)** for PR 28130 at commit [`537c594`](https://github.com/apache/spark/commit/537c594d27dc9b05719939d08af73f08d7e81e4d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #28160: [SPARK-30722][DOCS][FOLLOW-UP] Explicitly mention the same entire input/output length restriction of Series Iterator UDF
HyukjinKwon opened a new pull request #28160: [SPARK-30722][DOCS][FOLLOW-UP]
Explicitly mention the same entire input/output length restriction of Series
Iterator UDF
URL: https://github.com/apache/spark/pull/28160
### What changes were proposed in this pull request?
This PR explicitly mention that the requirement of Iterator of Series to
Iterator of Series and Iterator of Multiple Series to Iterator of Series
(previously Scalar Iterator pandas UDF).
The actual limitation of this UDF is the same length of the _entire input
and output_, instead of each series's length. Namely you can do something as
below:
```python
from typing import Iterator, Tuple
import pandas as pd
from pyspark.sql.functions import pandas_udf
@pandas_udf("long")
def func(
iterator: Iterator[pd.Series]) -> Iterator[pd.Series]:
return iter([pd.concat(iterator)])
spark.range(100).select(func("id")).show()
```
This characteristic allows you to prefetch the data from the iterator to
speed up, compared to the regular Scalar to Scalar (previously Scalar pandas
UDF).
### Why are the changes needed?
To document the correct restriction and characteristics of a feature.
### Does this PR introduce any user-facing change?
Yes in the documentation but only in unreleased branches.
### How was this patch tested?
Github Actions should test the documentation build
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27364: Adds support for Kubernetes NFS volume mounts.
dongjoon-hyun edited a comment on issue #27364: Adds support for Kubernetes NFS volume mounts. URL: https://github.com/apache/spark/pull/27364#issuecomment-611323182 Could you update the PR description by adding your comment and file an Apache Spark JIRA issue officially? It would be great if you can link K8s community issue or discussion about your requirement. > NFS could be used using PVC when we want to use some clean new empty disk space, but in order to use files in existing NFS shares, we need to use NFS volume mounts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27364: Adds support for Kubernetes NFS volume mounts.
dongjoon-hyun commented on issue #27364: Adds support for Kubernetes NFS volume mounts. URL: https://github.com/apache/spark/pull/27364#issuecomment-611323182 Could you update the PR description by adding your comment and file an Apache Spark JIRA issue officially? > NFS could be used using PVC when we want to use some clean new empty disk space, but in order to use files in existing NFS shares, we need to use NFS volume mounts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #28157: [SPARK-31390][SQL][DOCS] Document Window Function
huaxingao commented on a change in pull request #28157:
[SPARK-31390][SQL][DOCS] Document Window Function
URL: https://github.com/apache/spark/pull/28157#discussion_r405950823
##
File path: docs/sql-ref-functions-builtin-window.md
##
@@ -0,0 +1,215 @@
+---
+layout: global
+title: Window Functions
+displayTitle: Window Functions
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Similarly to aggregate functions, window functions operate on a group of rows.
However, unlike aggregate functions, window functions perform aggregation
without reducing, calculating a return value for each row in the group. Window
functions are useful for processing tasks such as calculating a moving average,
computing a cumulative, or accessing the value of rows given the relative
position of the current row. Spark SQL supports three types of window functions:
+ * Ranking Functions
+ * Analytic Functions
+ * Aggregate Functions
+
+### Types of Window Functions
+
+ Ranking Functions
+
+
+
+FunctionDescription
+
+
+ rank
+ Returns the rank of rows within a window partition. This is
equivalent to the RANK function in SQL.
+
+
+ dense_rank
+ Returns the rank of rows within a window partition, without any
gaps. This is equivalent to the DENSE_RANK function in SQL.
+
+
+ percent_rank
+ Returns the relative rank (i.e. percentile) of rows within a window
partition. This is equivalent to the PERCENT_RANK function in SQL.
+
+
+ ntile
+ Returns the ntile group id (from 1 to `n` inclusive) in an ordered
window partition. This is equivalent to the NTILE function in SQL.
+
+
+ row_number
+ Returns a sequential number starting from 1 within a window
partition. This is equivalent to the ROWNUMBER function in SQL.
+
+
+
+ Analytic Functions
+
+
+
+FunctionDescription
+
+
+ cume_dist
+ Returns the cumulative distribution of values within a window
partition, i.e. the fraction of rows that are below the current row. This is
equivalent to the CUMEDIST function in SQL.
+
+
+ lag
+ Returns the value that is offset rows before the
current row, and null if there are less than offset
rows before the current row.
+
+
+ lead
+ Returns the value that is offset rows after the current
row, and null if there are less than offset rows
after the current row. This is equivalent to the LEAD function in SQL.
+
+
+
+ Aggregate Functions
+
+Any of the aggregate functions can be used as a window function. Please refer
to the complete list of Spark [Aggregate
Functions](sql-ref-functions-builtin-aggregate.html).
+
+### How to Use Window Functions
+
+ * Mark a function as window function by using `over`.
+- SQL: Add an OVER clause after the window function, e.g. avg (...) OVER
(...);
+- DataFrame API: Call the window function's `over` method, e.g.
rank().over(...)
+ * Define the window specification associated with this function. A window
specification includes partitioning specification, ordering specification, and
frame specification.
+- Partitioning Specification:
+ - SQL: PARTITION BY
+ - DataFrame API: Window.partitionBy (...)
+- Ordering Specification:
+ - SQL: Order BY
+ - DataFrame API: Window.orderBy (...)
+- Frame Specification:
+ - SQL: ROWS (for ROW frame), RANGE (for RANGE frame)
+ - DataFrame API: WindowSpec.rowsBetween (for ROW frame),
WindowSpec.rangeBetween (for RANGE frame)
+
+### Examples
+
+{% highlight scala %}
+
+ import spark.implicits._
+
+ val data = Seq(("Lisa", "Sales", 1),
+("Evan", "Sales", 32000),
+("Fred", "Engineering", 21000),
+("Helen", "Marketing", 29000),
+("Alex", "Sales", 3),
+("Tom", "Engineering", 23000),
+("Jane", "Marketing", 29000),
+("Jeff", "Marketing", 35000),
+("Paul", "Engineering", 29000),
+("Chloe", "Engineering", 23000)
+ )
+ val df = data.toDF("name", "dept", "salary")
+ df.show()
+ // +-+---+--+
+ // | name| dept|salary|
+ // +-+---+--+
+ // | Lisa| Sales| 1|
+ // | Evan| Sales|
[GitHub] [spark] HyukjinKwon closed pull request #28135: [SPARK-26412][PYTHON][FOLLOW-UP] Improve error messages in Scala iterator pandas UDF
HyukjinKwon closed pull request #28135: [SPARK-26412][PYTHON][FOLLOW-UP] Improve error messages in Scala iterator pandas UDF URL: https://github.com/apache/spark/pull/28135 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28135: [SPARK-26412][PYTHON][FOLLOW-UP] Improve error messages in Scala iterator pandas UDF
HyukjinKwon commented on issue #28135: [SPARK-26412][PYTHON][FOLLOW-UP] Improve error messages in Scala iterator pandas UDF URL: https://github.com/apache/spark/pull/28135#issuecomment-611317766 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata
AmplabJenkins removed a comment on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata URL: https://github.com/apache/spark/pull/28147#issuecomment-611317284 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25688/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata
AmplabJenkins removed a comment on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata URL: https://github.com/apache/spark/pull/28147#issuecomment-611317281 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611317293 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25687/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611317288 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611317293 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25687/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata
AmplabJenkins commented on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata URL: https://github.com/apache/spark/pull/28147#issuecomment-611317281 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata
AmplabJenkins commented on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata URL: https://github.com/apache/spark/pull/28147#issuecomment-611317284 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25688/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611317288 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata
SparkQA commented on issue #28147: [SPARK-31357][SQL][WIP] Catalog API for view metadata URL: https://github.com/apache/spark/pull/28147#issuecomment-611316961 **[Test build #120997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120997/testReport)** for PR 28147 at commit [`3435379`](https://github.com/apache/spark/commit/3435379e01d9c088d2b3bc49c187c412fa7693bc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
SparkQA commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611316977 **[Test build #120996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120996/testReport)** for PR 28158 at commit [`9b54235`](https://github.com/apache/spark/commit/9b542356c672768a312a9ab10c79c0e33c86e5db). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
