[GitHub] [spark] AmplabJenkins commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675891086 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675891005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675891086 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins removed a comment on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675891005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675891103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32251/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675893141 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
SparkQA commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675893102 **[Test build #127624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127624/testReport)** for PR 29437 at commit [`6f147ee`](https://github.com/apache/spark/commit/6f147ee0ac20af8142a3dc715a9dbc7952f99265). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
SparkQA removed a comment on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675829698 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
AmplabJenkins commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675893281 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
AmplabJenkins commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675893267 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
SparkQA commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675893100 **[Test build #127622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127622/testReport)** for PR 29465 at commit [`84846a8`](https://github.com/apache/spark/commit/84846a873658fa609148e208f403d2e010e4114b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
AmplabJenkins commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675893274 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675893163 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
SparkQA commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675893114 **[Test build #127620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127620/testReport)** for PR 28953 at commit [`13f0dfc`](https://github.com/apache/spark/commit/13f0dfc2078f0933be01338a0fbf4d69113a9f4e). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
SparkQA commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675893101 **[Test build #127623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127623/testReport)** for PR 29469 at commit [`dd3e558`](https://github.com/apache/spark/commit/dd3e558073fb85d96f1ba096c231b04d86274fc4). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AlreadyPlanned(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
SparkQA commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675893104 **[Test build #127618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127618/testReport)** for PR 29434 at commit [`36a317a`](https://github.com/apache/spark/commit/36a317abca5a861a7b0c27317c80d030020838ed). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
SparkQA removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675890700 **[Test build #127627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127627/testReport)** for PR 29452 at commit [`9222f05`](https://github.com/apache/spark/commit/9222f05ca7d70cfa8795e78cef5f96024d8fdc0e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
SparkQA commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675893108 **[Test build #127626 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127626/testReport)** for PR 29474 at commit [`dd38f6d`](https://github.com/apache/spark/commit/dd38f6d1bf348e419108f88eac1b8207c9c792f0). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
SparkQA commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675893109 **[Test build #127627 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127627/testReport)** for PR 29452 at commit [`9222f05`](https://github.com/apache/spark/commit/9222f05ca7d70cfa8795e78cef5f96024d8fdc0e). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
SparkQA commented on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675893019 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos
SparkQA commented on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675893110 **[Test build #127619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127619/testReport)** for PR 29453 at commit [`b99ced4`](https://github.com/apache/spark/commit/b99ced462b8fd105a5a9fefd0ef6dc4e852d3257). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
AmplabJenkins removed a comment on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675893281 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos
SparkQA removed a comment on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675841543 **[Test build #127619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127619/testReport)** for PR 29453 at commit [`b99ced4`](https://github.com/apache/spark/commit/b99ced462b8fd105a5a9fefd0ef6dc4e852d3257). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675893163 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
SparkQA removed a comment on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675862392 **[Test build #127624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127624/testReport)** for PR 29437 at commit [`6f147ee`](https://github.com/apache/spark/commit/6f147ee0ac20af8142a3dc715a9dbc7952f99265). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
SparkQA removed a comment on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675862342 **[Test build #127623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127623/testReport)** for PR 29469 at commit [`dd3e558`](https://github.com/apache/spark/commit/dd3e558073fb85d96f1ba096c231b04d86274fc4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
SparkQA removed a comment on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675890659 **[Test build #127626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127626/testReport)** for PR 29474 at commit [`dd38f6d`](https://github.com/apache/spark/commit/dd38f6d1bf348e419108f88eac1b8207c9c792f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
AmplabJenkins removed a comment on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675893274 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins removed a comment on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675893141 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
SparkQA removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675858327 **[Test build #127622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127622/testReport)** for PR 29465 at commit [`84846a8`](https://github.com/apache/spark/commit/84846a873658fa609148e208f403d2e010e4114b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
SparkQA removed a comment on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675838022 **[Test build #127618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127618/testReport)** for PR 29434 at commit [`36a317a`](https://github.com/apache/spark/commit/36a317abca5a861a7b0c27317c80d030020838ed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675893553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
AmplabJenkins removed a comment on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675893267 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos
AmplabJenkins commented on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675893548 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
SparkQA removed a comment on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675844652 **[Test build #127620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127620/testReport)** for PR 28953 at commit [`13f0dfc`](https://github.com/apache/spark/commit/13f0dfc2078f0933be01338a0fbf4d69113a9f4e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
AmplabJenkins commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675893688 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
AmplabJenkins removed a comment on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675893688 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
AmplabJenkins removed a comment on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675893272 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127620/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
AmplabJenkins commented on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675893985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675893553 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins removed a comment on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675893147 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127626/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
AmplabJenkins removed a comment on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675893286 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127623/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
AmplabJenkins removed a comment on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675893283 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127624/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos
AmplabJenkins removed a comment on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675893548 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675893171 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127627/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
cloud-fan commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472786963
##
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] =
synchronized {
-hostToExecutors.get(host).map(_.toSet)
+
hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
Review comment:
It's super weird if `getExecutorsAliveOnHost` and
`hasExecutorsAliveOnHost` are not consistent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
AmplabJenkins removed a comment on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675893985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
cloud-fan commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472786561
##
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] =
synchronized {
-hostToExecutors.get(host).map(_.toSet)
+
hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
-hostToExecutors.contains(host)
+ hostToExecutors.get(host)
Review comment:
wrong indentation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675893566 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127622/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos
AmplabJenkins removed a comment on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675893555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127619/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
AmplabJenkins removed a comment on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675893692 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127618/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
AmplabJenkins removed a comment on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675894025 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127612/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
cloud-fan commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675895272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
cloud-fan commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675895620 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
cloud-fan commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675895746 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
SparkQA commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675896362 **[Test build #127628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127628/testReport)** for PR 29474 at commit [`dd38f6d`](https://github.com/apache/spark/commit/dd38f6d1bf348e419108f88eac1b8207c9c792f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
SparkQA commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675896435 **[Test build #127630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127630/testReport)** for PR 29437 at commit [`6f147ee`](https://github.com/apache/spark/commit/6f147ee0ac20af8142a3dc715a9dbc7952f99265). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
SparkQA commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675896403 **[Test build #127629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127629/testReport)** for PR 29452 at commit [`9222f05`](https://github.com/apache/spark/commit/9222f05ca7d70cfa8795e78cef5f96024d8fdc0e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
AmplabJenkins commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675896934 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675896851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675896811 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources
AmplabJenkins removed a comment on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675896934 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow
AmplabJenkins removed a comment on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675896811 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm
AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675896851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] manuzhang commented on a change in pull request #28032: [SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
manuzhang commented on a change in pull request #28032:
URL: https://github.com/apache/spark/pull/28032#discussion_r472799591
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##
@@ -229,6 +229,46 @@ case class DataSourceAnalysis(conf: SQLConf) extends
Rule[LogicalPlan] with Cast
}
}
+/**
+ * Add a repartition by dynamic partition columns before insert Datasource
table.
+ *
+ * Note that, this rule must be run after `DataSourceAnalysis`.
+ */
+case class RepartitionBeforeInsertDataSourceTable(conf: SQLConf) extends
Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+if (conf.repartitionBeforeInsert) {
+ insertRepartition(plan)
+} else {
+ plan
+}
+ }
+
+ private def insertRepartition(plan: LogicalPlan): LogicalPlan = plan
resolveOperators {
+case c @ CreateDataSourceTableAsSelectCommand(table, _, query, _)
+ if query.resolved && DDLUtils.isDatasourceTable(table) &&
table.bucketSpec.isEmpty
+&& table.partitionColumnNames.nonEmpty =>
+ val dynamicPartExps = table.partitionColumnNames.flatMap(n =>
query.output.find(_.name == n))
+ query match {
+case RepartitionByExpression(partExpressions, _, _) if partExpressions
== dynamicPartExps =>
+ c
+case _ =>
+ c.copy(query = RepartitionByExpression(dynamicPartExps, query,
conf.numShufflePartitions))
Review comment:
We can take advantage of AQE in case of data skew if we set
`optNumPartitions=None`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
SparkQA commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675902282 **[Test build #127632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127632/testReport)** for PR 28953 at commit [`62d7fc0`](https://github.com/apache/spark/commit/62d7fc0873ad4112c58e4783fbeffc09c1b5d2ec). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
SparkQA commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675902257 **[Test build #127631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127631/testReport)** for PR 29469 at commit [`87b1e53`](https://github.com/apache/spark/commit/87b1e53e9a2b2841cf420849eebb4bf20c6e8921). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
AmplabJenkins removed a comment on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675902670 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters
AmplabJenkins commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675902670 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
AmplabJenkins commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675902773 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC
AmplabJenkins removed a comment on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675902773 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
HyukjinKwon commented on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675904893 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
HyukjinKwon commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675905227 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
HyukjinKwon commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675905000 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
SparkQA commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675905337 **[Test build #127633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127633/testReport)** for PR 29465 at commit [`84846a8`](https://github.com/apache/spark/commit/84846a873658fa609148e208f403d2e010e4114b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
SparkQA commented on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675905373 **[Test build #127634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127634/testReport)** for PR 29460 at commit [`d029dba`](https://github.com/apache/spark/commit/d029dba3f72c1d94d9a1b560168c06f4ca21fd6d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675905938 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675905938 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
AmplabJenkins commented on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675906054 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
AmplabJenkins commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675906058 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0
AmplabJenkins removed a comment on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675906054 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
AmplabJenkins removed a comment on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675906058 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675907750 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
SparkQA commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675907718 **[Test build #127633 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127633/testReport)** for PR 29465 at commit [`84846a8`](https://github.com/apache/spark/commit/84846a873658fa609148e208f403d2e010e4114b). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675907750 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
SparkQA removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675905337 **[Test build #127633 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127633/testReport)** for PR 29465 at commit [`84846a8`](https://github.com/apache/spark/commit/84846a873658fa609148e208f403d2e010e4114b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4
AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675907758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127633/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
SparkQA commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675908858 **[Test build #127635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127635/testReport)** for PR 29434 at commit [`36a317a`](https://github.com/apache/spark/commit/36a317abca5a861a7b0c27317c80d030020838ed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
SparkQA commented on pull request #29468: URL: https://github.com/apache/spark/pull/29468#issuecomment-675914996 **[Test build #127636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127636/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
AmplabJenkins commented on pull request #29468: URL: https://github.com/apache/spark/pull/29468#issuecomment-675915430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
AmplabJenkins removed a comment on pull request #29468: URL: https://github.com/apache/spark/pull/29468#issuecomment-675915430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Ngone51 commented on pull request #29468: URL: https://github.com/apache/spark/pull/29468#issuecomment-675917559 > Now all we need is a real test to validate that the PR is actually fixing the delay scheduling issue you pointed out. @agrawaldevesh I added a unit test for locality level computation in `TaskSetManagerSuite`. I'm not sure what do you mean by a "real test". But if it means an end to end test which validates this fix can avoid the unnecessary delay, then I think it can be very hard. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
cloud-fan commented on a change in pull request #29434: URL: https://github.com/apache/spark/pull/29434#discussion_r472851122 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinCostBasedReorderSuite.scala ## @@ -329,7 +329,7 @@ class StarJoinCostBasedReorderSuite extends PlanTest with StatsEstimationTestBas // // Number of generated plans: 46 (vs. 82) val query = - d1.join(t3).join(t4).join(f1).join(d2).join(t5).join(t6).join(d3).join(t1).join(t2) + d1.join(t3).join(t4).join(f1).join(d3).join(d2).join(t5).join(t6).join(t1).join(t2) Review comment: what? the scala version changes the result of the join reorder rule? This is very weird as the join reorder is data-dependent and should be deterministic. also cc @wzhfy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29428: [SPARK-32608][SQL] Script Transform ROW FORMAT DELIMIT value should format value
cloud-fan commented on pull request #29428: URL: https://github.com/apache/spark/pull/29428#issuecomment-675943261 good catch! merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29428: [SPARK-32608][SQL] Script Transform ROW FORMAT DELIMIT value should format value
cloud-fan commented on pull request #29428: URL: https://github.com/apache/spark/pull/29428#issuecomment-675944589 @AngersZh can you open a new PR for 3.0? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29428: [SPARK-32608][SQL] Script Transform ROW FORMAT DELIMIT value should format value
cloud-fan closed pull request #29428: URL: https://github.com/apache/spark/pull/29428 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #29428: [SPARK-32608][SQL] Script Transform ROW FORMAT DELIMIT value should format value
AngersZh commented on pull request #29428: URL: https://github.com/apache/spark/pull/29428#issuecomment-675945277 > @AngersZh can you open a new PR for 3.0? Sure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29438: [SPARK-32607][SQL] Script Transformation ROW FORMAT DELIMITED `TOK_TABLEROWFORMATLINES` only support '\n'
cloud-fan commented on a change in pull request #29438:
URL: https://github.com/apache/spark/pull/29438#discussion_r472859934
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala
##
@@ -330,4 +330,42 @@ class SparkSqlParserSuite extends AnalysisTest {
assertEqual("ADD FILE /path with space/abc.txt", AddFileCommand("/path
with space/abc.txt"))
assertEqual("ADD JAR /path with space/abc.jar", AddJarCommand("/path with
space/abc.jar"))
}
+
+ test("SPARK-32607: Script Transformation ROW FORMAT DELIMITED" +
+" `TOK_TABLEROWFORMATLINES` only support '\\n'") {
+
+ // test input format TOK_TABLEROWFORMATLINES
+ intercept(
+ s"""
+ |SELECT TRANSFORM(a, b, c, d, e)
+ | ROW FORMAT DELIMITED
+ | FIELDS TERMINATED BY ','
+ | LINES TERMINATED BY '@'
+ | NULL DEFINED AS 'null'
+ | USING 'cat' AS (value)
+ | ROW FORMAT DELIMITED
+ | FIELDS TERMINATED BY '&'
+ | LINES TERMINATED BY '\n'
Review comment:
If it only supports one value, why do we ever provide a clause to set it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
LuciferYang commented on a change in pull request #29434: URL: https://github.com/apache/spark/pull/29434#discussion_r472861979 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinCostBasedReorderSuite.scala ## @@ -329,7 +329,7 @@ class StarJoinCostBasedReorderSuite extends PlanTest with StatsEstimationTestBas // // Number of generated plans: 46 (vs. 82) val query = - d1.join(t3).join(t4).join(f1).join(d2).join(t5).join(t6).join(d3).join(t1).join(t2) + d1.join(t3).join(t4).join(f1).join(d3).join(d2).join(t5).join(t6).join(t1).join(t2) Review comment: @cloud-fan Yep, in this case 2 candidates plan in level 4 has same `Cost`, From the code, we can see which candidate is generated first and which one we will choice. Related codes as follow: https://github.com/apache/spark/blob/3092527f7557b64ff9a5bedadfac8bb2f189a9b4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala#L209-L231 if `newJoinPlan` betterThan `existingPlan` will use `newJoinPlan` else use `existingPlan` and Same cost candidate not trigger update. In Scala 2.13 `HashMap` and `HashSet` has been rewritten, and I found the iteration order of `oneSideCandidates` and `otherSideCandidates` from `foundPlans` are changes through debug this case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13
LuciferYang commented on a change in pull request #29434: URL: https://github.com/apache/spark/pull/29434#discussion_r472861979 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinCostBasedReorderSuite.scala ## @@ -329,7 +329,7 @@ class StarJoinCostBasedReorderSuite extends PlanTest with StatsEstimationTestBas // // Number of generated plans: 46 (vs. 82) val query = - d1.join(t3).join(t4).join(f1).join(d2).join(t5).join(t6).join(d3).join(t1).join(t2) + d1.join(t3).join(t4).join(f1).join(d3).join(d2).join(t5).join(t6).join(t1).join(t2) Review comment: @cloud-fan Yep, in this case, 2 candidates plan in level 4 has same `Cost`, From the code, we can see which candidate is generated first and which one we will choice. Related codes as follow: https://github.com/apache/spark/blob/3092527f7557b64ff9a5bedadfac8bb2f189a9b4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala#L209-L231 if `newJoinPlan` betterThan `existingPlan` will use `newJoinPlan` else use `existingPlan` and Same cost candidate not trigger update. In Scala 2.13 `HashMap` and `HashSet` has been rewritten, and I found the iteration order of `oneSideCandidates` and `otherSideCandidates` from `foundPlans` are changes through debug this case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
