[GitHub] [spark] SparkQA commented on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
SparkQA commented on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-569885119 **[Test build #115981 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115981/testReport)** for PR 26993 at commit [`e6945e8`](https://github.com/apache/spark/commit/e6945e88a24d51551cba105b5e7e3825bc5e0a69). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins commented on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-569885283 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins commented on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-569885289 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115981/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
SparkQA removed a comment on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-569861048 **[Test build #115981 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115981/testReport)** for PR 26993 at commit [`e6945e8`](https://github.com/apache/spark/commit/e6945e88a24d51551cba105b5e7e3825bc5e0a69). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins removed a comment on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-569885283 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter
AmplabJenkins removed a comment on issue #26993: [SPARK-30338][SQL] Avoid unnecessary InternalRow copies in ParquetRowConverter URL: https://github.com/apache/spark/pull/26993#issuecomment-569885289 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115981/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on issue #27056: [SPARK-27217][SQL] Nested schema pruning with Aggregation
amanomer commented on issue #27056: [SPARK-27217][SQL] Nested schema pruning with Aggregation URL: https://github.com/apache/spark/pull/27056#issuecomment-569887119 cc @cloud-fan @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB commented on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table
07ARB commented on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table URL: https://github.com/apache/spark/pull/27047#issuecomment-569899785 @zhengruifeng , please merge this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB edited a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table
07ARB edited a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table URL: https://github.com/apache/spark/pull/27047#issuecomment-569899785 cc @zhengruifeng and @HyukjinKwon, please merge this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk opened a new pull request #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
MaxGekk opened a new pull request #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057 ### What changes were proposed in this pull request? In the PR, I propose to throw `AnalysisException` when a removed SQL config is set to non-default value. The following SQL configs removed by #26559 are marked as removed: 1. `spark.sql.fromJsonForceNullableSchema` 2. `spark.sql.legacy.compareDateTimestampInTimestamp` 3. `spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation` ### Why are the changes needed? To improve user experience with Spark SQL by notifying of removed SQL configs used by users. ### Does this PR introduce any user-facing change? Yes, before the `set` command was silently ignored: ```sql spark-sql> set spark.sql.fromJsonForceNullableSchema=false; spark.sql.fromJsonForceNullableSchemafalse ``` after the exception should be raised: ```sql spark-sql> set spark.sql.fromJsonForceNullableSchema=false; Error in query: The SQL config 'spark.sql.fromJsonForceNullableSchema' was removed in the version 3.0.0.; ``` ### How was this patch tested? Added new tests into `SQLConfSuite` for both cases when removed SQL configs are set to default and non-default values. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #26559: [SPARK-29930][SQL] Remove SQL configs declared to be removed in Spark 3.0
MaxGekk commented on a change in pull request #26559: [SPARK-29930][SQL] Remove
SQL configs declared to be removed in Spark 3.0
URL: https://github.com/apache/spark/pull/26559#discussion_r362186543
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -720,14 +720,6 @@ object SQLConf {
.stringConf
.createWithDefault("_corrupt_record")
- val FROM_JSON_FORCE_NULLABLE_SCHEMA =
buildConf("spark.sql.fromJsonForceNullableSchema")
Review comment:
Here is the PR https://github.com/apache/spark/pull/27057
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
SparkQA commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569902778 **[Test build #115984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115984/testReport)** for PR 27057 at commit [`73f887c`](https://github.com/apache/spark/commit/73f887cdfb3ae55e14d23200346099af481d9fab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569903060 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569903065 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20776/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569903060 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569903065 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20776/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB removed a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table
07ARB removed a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table URL: https://github.com/apache/spark/pull/27047#issuecomment-569899785 cc @zhengruifeng and @HyukjinKwon, please merge this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
cloud-fan commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569908126 sorry missed this one. Looks reasonable to me, can you reopen it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
cloud-fan commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569908659 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
SparkQA commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569909337 **[Test build #115985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115985/testReport)** for PR 26923 at commit [`125fa0f`](https://github.com/apache/spark/commit/125fa0f06cf5f4427d067b85a47ad3830c27c2a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
AmplabJenkins commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569909566 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569909568 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20778/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
AmplabJenkins commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569909570 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20777/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569909565 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
AmplabJenkins removed a comment on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569909570 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20777/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569909568 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20778/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
AmplabJenkins removed a comment on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569909566 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569909565 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569910529 **[Test build #115986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115986/testReport)** for PR 26813 at commit [`15f58ac`](https://github.com/apache/spark/commit/15f58ac0583c06c81bd4f6f63ddfe9da121b32a4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer opened a new pull request #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the u
beliefer opened a new pull request #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058 ### What changes were proposed in this pull request? This PR is related to https://github.com/apache/spark/pull/26656. https://github.com/apache/spark/pull/26656 only support use FILTER clause on aggregate expression without DISTINCT. This PR will enhance this feature when one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause. Such as: ``` select sum(distinct id) filter (where sex = 'man') from student; select class_id, sum(distinct id) filter (where sex = 'man') from student group by class_id; select count(id) filter (where class_id = 1), sum(distinct id) filter (where sex = 'man') from student; select class_id, count(id) filter (where class_id = 1), sum(distinct id) filter (where sex = 'man') from student group by class_id; select sum(distinct id), sum(distinct id) filter (where sex = 'man') from student; select class_id, sum(distinct id), sum(distinct id) filter (where sex = 'man') from student group by class_id; select class_id, count(id), count(id) filter (where class_id = 1), sum(distinct id), sum(distinct id) filter (where sex = 'man') from student group by class_id; ``` This PR not support: ``` select class_id, count(distinct sex), sum(distinct id) filter (where sex = 'man') from student group by class_id; select class_id, count(distinct sex) filter (where class_id = 1), sum(distinct id) filter (where sex = 'man') from student group by class_id; ``` https://issues.apache.org/jira/browse/SPARK-30396 used for later. ### Why are the changes needed? No ### Does this PR introduce any user-facing change? No ### How was this patch tested? New UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of th
SparkQA commented on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#issuecomment-569911830 **[Test build #115987 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115987/testReport)** for PR 27058 at commit [`0008bae`](https://github.com/apache/spark/commit/0008bae5ff27a6766b9ceb3bb067eac2e8e322dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB commented on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table
07ARB commented on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table URL: https://github.com/apache/spark/pull/27047#issuecomment-569911911 @HyukjinKwon and @zhengruifeng , can you please review this MR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows
AmplabJenkins removed a comment on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#issuecomment-569912072 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20779/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use
AmplabJenkins commented on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#issuecomment-569912064 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows
AmplabJenkins removed a comment on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#issuecomment-569912064 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use
AmplabJenkins commented on issue #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#issuecomment-569912072 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20779/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
SparkQA commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569914273 **[Test build #115988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115988/testReport)** for PR 27059 at commit [`e34c4fb`](https://github.com/apache/spark/commit/e34c4fb862a4fc1f3320705c4385dca866ae46fd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng opened a new pull request #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
zhengruifeng opened a new pull request #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059 ### What changes were proposed in this pull request? use `.ml.Summarizer` instead of `.mllib.MultivariateOnlineSummarizer` to avoid computation of unused metrics ### Why are the changes needed? to avoid computation of unused metrics ### Does this PR introduce any user-facing change? No ### How was this patch tested? existing testsuites This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569914430 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569914435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20780/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569914435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20780/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569914430 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB edited a comment on issue #27051: [SPARK-30389][SQL]Validate file type extension during add jar command.
07ARB edited a comment on issue #27051: [SPARK-30389][SQL]Validate file type extension during add jar command. URL: https://github.com/apache/spark/pull/27051#issuecomment-569855412 even if we are doing like this, i feel it's not correct as per our documentation, we should force end-user to upload proper extension file. https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-aux-resource-mgmt-add-jar.html  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 012huang opened a new pull request #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak
012huang opened a new pull request #27060: [SPARK-30246][CORE][SHUFFLE]fix
spark external shuffle memory leak
URL: https://github.com/apache/spark/pull/27060
### What changes were proposed in this pull request?
An app finished abnormal sometimes may cause shuffe service memory leak. In
one of our production cases, the app failed for Stage cancelled as SparkContext
has already shut down. the strange is there are still requests for fetch
shuffle data and cause error in server side as below:
```
2019-12-08 22:23:33,375 ERROR server.TransportRequestHandler
(TransportRequestHandler.java:processFetchRequest(132)) - Error opening block
StreamChunkId{streamId=1902064894814, chunkIndex=0} for request from
/10.221.115.175:38582
java.lang.RuntimeException: Executor is not registered
(appId=application_1574499669561_954327, execId=4514)
```
the client sie also show corresponding log like this:
```
org.apache.spark.shuffle.FetchFailedException: Failure while fetching
StreamChunkId{streamId=1902064894814, chunkIndex=0}:
java.lang.RuntimeException: Executor is not registered
(appId=application_1574499669561_954327, execId=4514)
```
in some cases, the request for `OpenBlocks` is still on the fly. In the code
`ExternalShuffleBlockHandler#handleMessage`, it will register a `StreamState`
to `OneForOneStreamManager#streams`, then reply an success response to client
unconditionally , the client receive the response and then fire
`ChunkFetchRequest` to fetch chunk, but at this time, the app has got event
`APPLICATION_STOP` and executed `ExternalShuffleService#applicationRemoved`
method to clean the app's `ExecutorShuffleInfo`, this made `Executor is not
registered` error happended. even though when the client channel is closing,
the `TransportRequestHandler#channelInactive` was called to clean the
StreamState with relate channel, but when cleanning the `StreamState buffter`,
it also lookup `ManagedBuffer` with` appId` and `execId` info which have been
cleaned in executors object. we can also find the log: `StreamManager
connectionTerminated() callback failed` in NM's log file.
so, when an `OpenBlocks` request come, we should lookup
`ExternalShuffleBlockResolver#executors` , if the realted app is exited, we
should not registering a `StreamState` and just close the client (or reply an
special message to client and in client side to handle it). and when an app get
`APPLICATION_STOP` to call `applicationRemoved`, we should clean the the
related `streamState` before `ExecutorShuffleInfo` has been cleaned, this is
what the PR changes and prevents the shuffle service memory leak.
### Why are the changes needed?
The external shuffle service memory leak has a great impact on cluster with
dynanic on and may cause NM crash.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
with existing ut
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak
AmplabJenkins commented on issue #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak URL: https://github.com/apache/spark/pull/27060#issuecomment-569916413 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak
AmplabJenkins commented on issue #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak URL: https://github.com/apache/spark/pull/27060#issuecomment-569916599 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak
AmplabJenkins removed a comment on issue #27060: [SPARK-30246][CORE][SHUFFLE]fix spark external shuffle memory leak URL: https://github.com/apache/spark/pull/27060#issuecomment-569916413 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569918508 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115986/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569918504 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569918504 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
SparkQA removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569910529 **[Test build #115986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115986/testReport)** for PR 26813 at commit [`15f58ac`](https://github.com/apache/spark/commit/15f58ac0583c06c81bd4f6f63ddfe9da121b32a4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569918463 **[Test build #115986 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115986/testReport)** for PR 26813 at commit [`15f58ac`](https://github.com/apache/spark/commit/15f58ac0583c06c81bd4f6f63ddfe9da121b32a4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569918508 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115986/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
cloud-fan commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-569920054 @AngersZh any updates? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
cloud-fan commented on a change in pull request #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#discussion_r362206963 ## File path: sql/core/src/test/resources/sql-tests/inputs/explain.sql ## @@ -4,7 +4,7 @@ CREATE table explain_temp2 (key int, val int) USING PARQUET; CREATE table explain_temp3 (key int, val int) USING PARQUET; SET spark.sql.codegen.wholeStage = true; - +SET spark.sql.adaptive.enabled = false; Review comment: since we are touchinng it, let's use the right approach. We should put this line at the beginning of this file ``` --SET spark.sql.codegen.wholeStage = true --SET spark.sql.adaptive.enabled = false ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
cloud-fan commented on a change in pull request #26813: [SPARK-30188][SQL][WIP]
Enable adaptive query execution by default
URL: https://github.com/apache/spark/pull/26813#discussion_r362207621
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
##
@@ -89,38 +91,46 @@ class DataFrameCallbackSuite extends QueryTest with
SharedSparkSession {
}
test("get numRows metrics by callback") {
-val metrics = ArrayBuffer.empty[Long]
-val listener = new QueryExecutionListener {
- // Only test successful case here, so no need to implement `onFailure`
- override def onFailure(funcName: String, qe: QueryExecution, error:
Throwable): Unit = {}
-
- override def onSuccess(funcName: String, qe: QueryExecution, duration:
Long): Unit = {
-val metric = qe.executedPlan match {
- case w: WholeStageCodegenExec => w.child.longMetric("numOutputRows")
- case other => other.longMetric("numOutputRows")
+withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
+ // with AQE on, the WholeStageCodegen rule is applied when running
QueryStageExec.
+ val metrics = ArrayBuffer.empty[Long]
+ val listener = new QueryExecutionListener {
+// Only test successful case here, so no need to implement `onFailure`
+override def onFailure(funcName: String, qe: QueryExecution, error:
Throwable): Unit = {}
+
+override def onSuccess(funcName: String, qe: QueryExecution, duration:
Long): Unit = {
+ val plan = qe.executedPlan match {
+case a: AdaptiveSparkPlanExec => a.executedPlan
Review comment:
This appears many times, we can introduce a `stripAQEPlan` method in
`AdaptiveSparkPlanHelper` to do it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
SparkQA commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569922368 **[Test build #115988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115988/testReport)** for PR 27059 at commit [`e34c4fb`](https://github.com/apache/spark/commit/e34c4fb862a4fc1f3320705c4385dca866ae46fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
SparkQA removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569914273 **[Test build #115988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115988/testReport)** for PR 27059 at commit [`e34c4fb`](https://github.com/apache/spark/commit/e34c4fb862a4fc1f3320705c4385dca866ae46fd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569922483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115988/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins commented on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569922479 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569922483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115988/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation
AmplabJenkins removed a comment on issue #27059: [SPARK-30398][ML] PCA/RegressionMetrics/RowMatrix avoid unnecessary computation URL: https://github.com/apache/spark/pull/27059#issuecomment-569922479 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB edited a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table
07ARB edited a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table URL: https://github.com/apache/spark/pull/27047#issuecomment-569911911 cc @srowen all tests passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569924868 **[Test build #115989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115989/testReport)** for PR 26813 at commit [`2f9307e`](https://github.com/apache/spark/commit/2f9307e28e6aaf1bc51226879cbbebbb08d95502). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569925070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20781/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569925066 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569925070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20781/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569925066 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu opened a new pull request #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
fuwhu opened a new pull request #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850 ### What changes were proposed in this pull request? Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex. ### Why are the changes needed? PrunedInMemoryFileIndex is only used in CatalogFileIndex.filterPartitions, and its name is kind of confusing, we can completely merge its functionality into InMemoryFileIndex and remove the class. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
AmplabJenkins commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569926171 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
AmplabJenkins commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569926173 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20782/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
AmplabJenkins removed a comment on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569926173 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20782/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
fuwhu commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569926243 @cloud-fan Sure, reopened, please help review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
AmplabJenkins removed a comment on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569926171 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex
SparkQA commented on issue #26850: [SPARK-30215][SQL] Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex URL: https://github.com/apache/spark/pull/26850#issuecomment-569927042 **[Test build #115990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115990/testReport)** for PR 26850 at commit [`a2d59ca`](https://github.com/apache/spark/commit/a2d59ca27f84ce040cdb8908cbad76266d749f10). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] 07ARB edited a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table
07ARB edited a comment on issue #27047: [SPARK-30384][WEBUI]Needs to improve the Column name and Add tooltips for the Fair Scheduler Pool Table URL: https://github.com/apache/spark/pull/27047#issuecomment-569911911 cc @srowen all tests passed. Wish you a Happy New Year. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ConeyLiu closed pull request #24237: [SPARK-27319][SQL] Filter out dir based on PathFilter before listing them
ConeyLiu closed pull request #24237: [SPARK-27319][SQL] Filter out dir based on PathFilter before listing them URL: https://github.com/apache/spark/pull/24237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 opened a new pull request #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs
Ngone51 opened a new pull request #27061: [MINOR][CORE] Process bar should
print new line to avoid polluting logs
URL: https://github.com/apache/spark/pull/27061
### What changes were proposed in this pull request?
Use `println()` instead of `print()` to show process bar in console.
### Why are the changes needed?
Logs are polluted by process bar:
This is easy to reproduce:
1. start `./bin/spark-shell`
2. `sc.setLogLevel("INFO")`
3. run: `spark.range(1).coalesce(1).write.parquet("/tmp/result")`
### Does this PR introduce any user-facing change?
Yeah, more friendly format in console.
### How was this patch tested?
Tested manually.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs
AmplabJenkins commented on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs URL: https://github.com/apache/spark/pull/27061#issuecomment-569930715 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20783/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs
AmplabJenkins commented on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs URL: https://github.com/apache/spark/pull/27061#issuecomment-569930711 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs
SparkQA commented on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs URL: https://github.com/apache/spark/pull/27061#issuecomment-569931641 **[Test build #115991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115991/testReport)** for PR 27061 at commit [`91217ef`](https://github.com/apache/spark/commit/91217ef1e3ae1c60d12cd0eaa6377cbf700304e3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs
AmplabJenkins removed a comment on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs URL: https://github.com/apache/spark/pull/27061#issuecomment-569930711 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs
AmplabJenkins removed a comment on issue #27061: [MINOR][CORE] Process bar should print new line to avoid polluting logs URL: https://github.com/apache/spark/pull/27061#issuecomment-569930715 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20783/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer edited a comment on issue #27056: [SPARK-27217][SQL] Nested schema pruning with Aggregation
amanomer edited a comment on issue #27056: [SPARK-27217][SQL] Nested schema pruning with Aggregation URL: https://github.com/apache/spark/pull/27056#issuecomment-569887119 cc @cloud-fan @HyukjinKwon @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
SparkQA commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569933577 **[Test build #115984 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115984/testReport)** for PR 27057 at commit [`73f887c`](https://github.com/apache/spark/commit/73f887cdfb3ae55e14d23200346099af481d9fab). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
SparkQA removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569902778 **[Test build #115984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115984/testReport)** for PR 27057 at commit [`73f887c`](https://github.com/apache/spark/commit/73f887cdfb3ae55e14d23200346099af481d9fab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569933887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115984/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569933882 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins commented on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569933887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115984/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs
AmplabJenkins removed a comment on issue #27057: [SPARK-29930][SQL][FOLLOW-UP] Allow only default value to be set for removed SQL configs URL: https://github.com/apache/spark/pull/27057#issuecomment-569933882 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569937969 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115989/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569937964 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
SparkQA commented on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569937916 **[Test build #115989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115989/testReport)** for PR 26813 at commit [`2f9307e`](https://github.com/apache/spark/commit/2f9307e28e6aaf1bc51226879cbbebbb08d95502). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
SparkQA removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569924868 **[Test build #115989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115989/testReport)** for PR 26813 at commit [`2f9307e`](https://github.com/apache/spark/commit/2f9307e28e6aaf1bc51226879cbbebbb08d95502). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569937964 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default
AmplabJenkins removed a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-569937969 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115989/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression
beliefer commented on a change in pull request #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#discussion_r362226499 ## File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out ## @@ -177,43 +185,56 @@ struct NULL NULL --- !query 16 +-- !query 17 +SELECT dept_id, SUM(DISTINCT salary) FILTER (WHERE date_format(hiredate, "-MM-dd HH:mm:ss") > "2001-01-01 00:00:00") FROM emp GROUP BY dept_id +-- !query 17 schema +struct +-- !query 17 output +10 300.0 Review comment: This result is incorrect. I will check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression
beliefer commented on a change in pull request #27058: [SPARK-30395][SQL] When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause URL: https://github.com/apache/spark/pull/27058#discussion_r362227598 ## File path: sql/core/src/test/resources/sql-tests/results/group-by-filter.sql.out ## @@ -95,41 +95,49 @@ struct -- !query 9 -SELECT a, COUNT(b) FILTER (WHERE a >= 2) FROM testData GROUP BY a +SELECT COUNT(DISTINCT id) FILTER (WHERE date_format(hiredate, "-MM-dd HH:mm:ss") = "2001-01-01 00:00:00") FROM emp -- !query 9 schema -struct +struct -- !query 9 output +3 Review comment: This result is incorrect too. It seems the I have to check `date_format`. @maropu Do you know the reason? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
SparkQA removed a comment on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569909337 **[Test build #115985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115985/testReport)** for PR 26923 at commit [`125fa0f`](https://github.com/apache/spark/commit/125fa0f06cf5f4427d067b85a47ad3830c27c2a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace
SparkQA commented on issue #26923: [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and namespace URL: https://github.com/apache/spark/pull/26923#issuecomment-569940790 **[Test build #115985 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115985/testReport)** for PR 26923 at commit [`125fa0f`](https://github.com/apache/spark/commit/125fa0f06cf5f4427d067b85a47ad3830c27c2a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
