[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533774243 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/10/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533774240 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
SparkQA removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768110 **[Test build #10 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/10/testReport)** for PR 25294 at commit [`2a95580`](https://github.com/apache/spark/commit/2a95580b985dfcf43e2c4d3336ce0286b7c245c0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533774240 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
SparkQA commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533774207 **[Test build #10 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/10/testReport)** for PR 25294 at commit [`2a95580`](https://github.com/apache/spark/commit/2a95580b985dfcf43e2c4d3336ce0286b7c245c0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533774243 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/10/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest
AmplabJenkins removed a comment on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest URL: https://github.com/apache/spark/pull/25788#issuecomment-533773958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/04/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest
AmplabJenkins removed a comment on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest URL: https://github.com/apache/spark/pull/25788#issuecomment-533773955 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest
AmplabJenkins commented on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest URL: https://github.com/apache/spark/pull/25788#issuecomment-533773955 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest
AmplabJenkins commented on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest URL: https://github.com/apache/spark/pull/25788#issuecomment-533773958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/04/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest
SparkQA commented on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest URL: https://github.com/apache/spark/pull/25788#issuecomment-533773835 **[Test build #04 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/04/testReport)** for PR 25788 at commit [`9f32999`](https://github.com/apache/spark/commit/9f329992a7baec1ae04033b2e75b47f7210fb94f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest
SparkQA removed a comment on issue #25788: [SPARK-29084][SQL][TESTS] Check method bytecode size in BenchmarkQueryTest URL: https://github.com/apache/spark/pull/25788#issuecomment-533761615 **[Test build #04 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/04/testReport)** for PR 25788 at commit [`9f32999`](https://github.com/apache/spark/commit/9f329992a7baec1ae04033b2e75b47f7210fb94f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533773662 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/07/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533773661 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533773661 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533773662 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/07/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
SparkQA removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533766701 **[Test build #07 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/07/testReport)** for PR 25879 at commit [`9a48846`](https://github.com/apache/spark/commit/9a4884674ea321af896985c8e9e34721ad989ec1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
SparkQA commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533773551 **[Test build #07 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/07/testReport)** for PR 25879 at commit [`9a48846`](https://github.com/apache/spark/commit/9a4884674ea321af896985c8e9e34721ad989ec1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
AmplabJenkins commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533773119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/06/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
AmplabJenkins removed a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533773118 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
AmplabJenkins commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533773118 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
AmplabJenkins removed a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533773119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/06/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
SparkQA removed a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533764872 **[Test build #06 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/06/testReport)** for PR 25845 at commit [`c39b06f`](https://github.com/apache/spark/commit/c39b06f4b305a094b4c132b1f6dee5dc4332a1d8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
SparkQA commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533773003 **[Test build #06 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/06/testReport)** for PR 25845 at commit [`c39b06f`](https://github.com/apache/spark/commit/c39b06f4b305a094b4c132b1f6dee5dc4332a1d8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25880: Correct a log message
AmplabJenkins commented on issue #25880: Correct a log message URL: https://github.com/apache/spark/pull/25880#issuecomment-533772547 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25880: Correct a log message
AmplabJenkins removed a comment on issue #25880: Correct a log message URL: https://github.com/apache/spark/pull/25880#issuecomment-533772490 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25880: Correct a log message
AmplabJenkins commented on issue #25880: Correct a log message URL: https://github.com/apache/spark/pull/25880#issuecomment-533772490 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mdianjun opened a new pull request #25880: Correct a log message
mdianjun opened a new pull request #25880: Correct a log message URL: https://github.com/apache/spark/pull/25880 ### What changes were proposed in this pull request? Correct a word in a log message. ### Why are the changes needed? Log message will be more clearly. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Test is not needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768183 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768185 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16191/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768185 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16191/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768183 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
SparkQA commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768110 **[Test build #10 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/10/testReport)** for PR 25294 at commit [`2a95580`](https://github.com/apache/spark/commit/2a95580b985dfcf43e2c4d3336ce0286b7c245c0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25746: [WIP][SPARK-28292][SQL] Enable Injection of User-defined Hint
dongjoon-hyun commented on issue #25746: [WIP][SPARK-28292][SQL] Enable Injection of User-defined Hint URL: https://github.com/apache/spark/pull/25746#issuecomment-533768107 Got it. Thank you for making this change, @gatorsmile ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
wangyum commented on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-533768021 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result.
AmplabJenkins removed a comment on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result. URL: https://github.com/apache/spark/pull/25863#issuecomment-533767809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16190/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result.
AmplabJenkins removed a comment on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result. URL: https://github.com/apache/spark/pull/25863#issuecomment-533767808 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result.
AmplabJenkins commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result. URL: https://github.com/apache/spark/pull/25863#issuecomment-533767809 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16190/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result.
AmplabJenkins commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result. URL: https://github.com/apache/spark/pull/25863#issuecomment-533767808 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result.
SparkQA commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result. URL: https://github.com/apache/spark/pull/25863#issuecomment-533767738 **[Test build #09 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/09/testReport)** for PR 25863 at commit [`a4f33d9`](https://github.com/apache/spark/commit/a4f33d97aa2747973eb39e0218ac42b591e8b473). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25877: [NO-NOT-MERGE][test-maven] Verify timeout for maven SparkPullRequestBuilder
dongjoon-hyun commented on issue #25877: [NO-NOT-MERGE][test-maven] Verify timeout for maven SparkPullRequestBuilder URL: https://github.com/apache/spark/pull/25877#issuecomment-533767588 @wangyum . Do you need to increase this to pass your https://github.com/apache/spark/pull/25690? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result.
turboFei commented on issue #25863: [WIP][SPARK-29037][CORE][SQL] For static partition overwrite, spark may give duplicate result. URL: https://github.com/apache/spark/pull/25863#issuecomment-533767506 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
dongjoon-hyun edited a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767464 `Linters` job passed in `9 min` already and this PR is irrelevant to Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
dongjoon-hyun commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767464 `Linters` job passed in `9 min` already. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767093 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16189/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767092 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767093 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16189/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767092 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions
AmplabJenkins removed a comment on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions URL: https://github.com/apache/spark/pull/25827#issuecomment-533767013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111099/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions
AmplabJenkins removed a comment on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions URL: https://github.com/apache/spark/pull/25827#issuecomment-533767011 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
SparkQA commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533767026 **[Test build #08 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/08/testReport)** for PR 25879 at commit [`e6318df`](https://github.com/apache/spark/commit/e6318dfbdae2898c4745fdc6fa8a30f02753ab9f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions
AmplabJenkins commented on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions URL: https://github.com/apache/spark/pull/25827#issuecomment-533767011 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions
AmplabJenkins commented on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions URL: https://github.com/apache/spark/pull/25827#issuecomment-533767013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111099/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions
SparkQA removed a comment on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions URL: https://github.com/apache/spark/pull/25827#issuecomment-533747209 **[Test build #111099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111099/testReport)** for PR 25827 at commit [`ca6522a`](https://github.com/apache/spark/commit/ca6522ab1ed7edbeaf46a306de3de8d5828a528a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions
SparkQA commented on issue #25827: [SPARK-29128][SQL] Split predicate code in OR expressions URL: https://github.com/apache/spark/pull/25827#issuecomment-533766923 **[Test build #111099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111099/testReport)** for PR 25827 at commit [`ca6522a`](https://github.com/apache/spark/commit/ca6522ab1ed7edbeaf46a306de3de8d5828a528a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533766766 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16188/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins removed a comment on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533766764 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533766766 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16188/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
AmplabJenkins commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533766764 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
SparkQA commented on issue #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879#issuecomment-533766701 **[Test build #07 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/07/testReport)** for PR 25879 at commit [`9a48846`](https://github.com/apache/spark/commit/9a4884674ea321af896985c8e9e34721ad989ec1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
AmplabJenkins removed a comment on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x)) URL: https://github.com/apache/spark/pull/25878#issuecomment-533766427 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
AmplabJenkins commented on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x)) URL: https://github.com/apache/spark/pull/25878#issuecomment-533766680 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action
dongjoon-hyun opened a new pull request #25879: [SPARK-29199][INFRA] Add linters and license/dependency checkers to GitHub Action URL: https://github.com/apache/spark/pull/25879 ### What changes were proposed in this pull request? This PR aims to add linters and license/dependency checkers to GitHub Action. ### Why are the changes needed? This will help the PR reviews. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? See the GitHub Action result on this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
AmplabJenkins commented on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x)) URL: https://github.com/apache/spark/pull/25878#issuecomment-533766427 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
AngersZh commented on issue #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x)) URL: https://github.com/apache/spark/pull/25878#issuecomment-533766516 gentle ping @juliuszsompolski I agree a lot that we should simplify these expression. It's really useful for judge query canonicalization equal. Meet this problem when I am doing some cache framework . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x))
AngersZh opened a new pull request #25878: [SPARK-29162][SQL]Simplify NOT(IsNull(x)) and NOT(IsNotNull(x)) URL: https://github.com/apache/spark/pull/25878 ### What changes were proposed in this pull request? Rewrite ``` NOT isnull(x) -> isnotnull(x) NOT isnotnull(x) -> isnull(x) ``` ### Why are the changes needed? Make LogicalPlan more readable and useful for query canonicalization. Make same condition equal when judge query canonicalization equal ### Does this PR introduce any user-facing change? NO ### How was this patch tested? UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
AmplabJenkins commented on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore URL: https://github.com/apache/spark/pull/25811#issuecomment-533764926 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/02/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
AmplabJenkins commented on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore URL: https://github.com/apache/spark/pull/25811#issuecomment-533764924 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
AmplabJenkins removed a comment on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore URL: https://github.com/apache/spark/pull/25811#issuecomment-533764926 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/02/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
AmplabJenkins removed a comment on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore URL: https://github.com/apache/spark/pull/25811#issuecomment-533764924 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
SparkQA commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533764872 **[Test build #06 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/06/testReport)** for PR 25845 at commit [`c39b06f`](https://github.com/apache/spark/commit/c39b06f4b305a094b4c132b1f6dee5dc4332a1d8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
SparkQA removed a comment on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore URL: https://github.com/apache/spark/pull/25811#issuecomment-533756236 **[Test build #02 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/02/testReport)** for PR 25811 at commit [`b6e65f8`](https://github.com/apache/spark/commit/b6e65f84eaba813217bbae1264d3560da7c677e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore
SparkQA commented on issue #25811: [SPARK-29111][CORE] Support snapshot/restore on KVStore URL: https://github.com/apache/spark/pull/25811#issuecomment-533764838 **[Test build #02 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/02/testReport)** for PR 25811 at commit [`b6e65f8`](https://github.com/apache/spark/commit/b6e65f84eaba813217bbae1264d3560da7c677e9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
dongjoon-hyun commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533764555 Ya. I agree with you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
HeartSaVioR edited a comment on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533764134 Yeah I thought someone may say it should worth to provide backward-compatible option so waited for couple of days to see more voices, but doesn't look like so. I'll mention the change to the migration note. Thanks for reminding! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file
HeartSaVioR commented on issue #25845: [SPARK-29160][CORE] Use UTF-8 explicitly for reading/writing event log file URL: https://github.com/apache/spark/pull/25845#issuecomment-533764134 Yeah I thought someone may say it should worth to provide backward-compatible option so waited for couple of days to see more voices, but doesn't look like so. I'll mention the change to the migration note. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326845315 ## File path: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPoolSuite.scala ## @@ -29,7 +29,14 @@ import org.apache.spark.SparkConf import org.apache.spark.sql.kafka010.KafkaDataConsumer.CacheKey import org.apache.spark.sql.test.SharedSparkSession -class InternalKafkaConsumerPoolSuite extends SharedSparkSession { + Review comment: nit: remove this line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844878 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPool.scala ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} +import java.util.concurrent.ConcurrentHashMap + +import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, PooledObject, SwallowedExceptionListener} +import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig} + +import org.apache.spark.internal.Logging + +/** + * Provides object pool for objects which is grouped by a key. + * + * This class leverages [[GenericKeyedObjectPool]] internally, hence providing methods based on + * the class, and same contract applies: after using the borrowed object, you must either call + * returnObject() if the object is healthy to return to pool, or invalidateObject() if the object + * should be destroyed. + * + * The soft capacity of pool is determined by "poolConfig.capacity" config value, + * and the pool will have reasonable default value if the value is not provided. + * (The instance will do its best effort to respect soft capacity but it can exceed when there's + * a borrowing request and there's neither free space nor idle object to clear.) + * + * This class guarantees that no caller will get pooled object once the object is borrowed and + * not yet returned, hence provide thread-safety usage of non-thread-safe objects unless caller + * shares the object to multiple threads. + */ +private[kafka010] abstract class InternalKafkaConnectorPool[K, V]( +objectFactory: ObjectFactory[K, V], +poolConfig: PoolConfig[V], +swallowedExceptionListener: SwallowedExceptionListener) extends Logging { + + // the class is intended to have only soft capacity + assert(poolConfig.getMaxTotal < 0) + + private val pool = { +val internalPool = new GenericKeyedObjectPool[K, V](objectFactory, poolConfig) +internalPool.setSwallowedExceptionListener(swallowedExceptionListener) +internalPool + } + + /** + * Borrows object from the pool. If there's no idle object for the key, + * the pool will create the object. + * + * If the pool doesn't have idle object for the key and also exceeds the soft capacity, + * pool will try to clear some of idle objects. + * + * Borrowed object must be returned by either calling returnObject or invalidateObject, otherwise + * the object will be kept in pool as active object. + */ + def borrowObject(key: K, kafkaParams: ju.Map[String, Object]): V = { +updateKafkaParamForKey(key, kafkaParams) + +if (size >= poolConfig.softMaxSize) { + logWarning("Pool exceeds its soft max size, cleaning up idle objects...") + pool.clearOldest() +} + +pool.borrowObject(key) + } + + /** Returns borrowed object to the pool. */ + def returnObject(connector: V): Unit = { +pool.returnObject(createKey(connector), connector) + } + + /** Invalidates (destroy) borrowed object to the pool. */ + def invalidateObject(connector: V): Unit = { +pool.invalidateObject(createKey(connector), connector) + } + + /** Invalidates all idle values for the key */ + def invalidateKey(key: K): Unit = { +pool.clear(key) + } + + /** + * Closes the keyed object pool. Once the pool is closed, + * borrowObject will fail with [[IllegalStateException]], but returnObject and invalidateObject + * will continue to work, with returned objects destroyed on return. + * + * Also destroys idle instances in the pool. + */ + def close(): Unit = { +pool.close() + } + + def reset(): Unit = { +// this is the best-effort of clearing up. otherwise we should close the pool and create again +// but we don't want to make it "var" only because of tests. +pool.clear() + } + + def numIdle: Int = pool.getNumIdle + + def numIdle(key: K): Int = pool.getNumIdle(key) + + def numActive: Int = pool.getNumA
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844478 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPool.scala ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} +import java.util.concurrent.ConcurrentHashMap + +import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, PooledObject, SwallowedExceptionListener} +import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig} + +import org.apache.spark.internal.Logging + +/** + * Provides object pool for objects which is grouped by a key. + * + * This class leverages [[GenericKeyedObjectPool]] internally, hence providing methods based on + * the class, and same contract applies: after using the borrowed object, you must either call + * returnObject() if the object is healthy to return to pool, or invalidateObject() if the object + * should be destroyed. + * + * The soft capacity of pool is determined by "poolConfig.capacity" config value, + * and the pool will have reasonable default value if the value is not provided. + * (The instance will do its best effort to respect soft capacity but it can exceed when there's + * a borrowing request and there's neither free space nor idle object to clear.) + * + * This class guarantees that no caller will get pooled object once the object is borrowed and + * not yet returned, hence provide thread-safety usage of non-thread-safe objects unless caller + * shares the object to multiple threads. + */ +private[kafka010] abstract class InternalKafkaConnectorPool[K, V]( +objectFactory: ObjectFactory[K, V], +poolConfig: PoolConfig[V], +swallowedExceptionListener: SwallowedExceptionListener) extends Logging { + + // the class is intended to have only soft capacity + assert(poolConfig.getMaxTotal < 0) + + private val pool = { +val internalPool = new GenericKeyedObjectPool[K, V](objectFactory, poolConfig) +internalPool.setSwallowedExceptionListener(swallowedExceptionListener) +internalPool + } + + /** + * Borrows object from the pool. If there's no idle object for the key, + * the pool will create the object. + * + * If the pool doesn't have idle object for the key and also exceeds the soft capacity, + * pool will try to clear some of idle objects. + * + * Borrowed object must be returned by either calling returnObject or invalidateObject, otherwise + * the object will be kept in pool as active object. + */ + def borrowObject(key: K, kafkaParams: ju.Map[String, Object]): V = { +updateKafkaParamForKey(key, kafkaParams) + +if (size >= poolConfig.softMaxSize) { + logWarning("Pool exceeds its soft max size, cleaning up idle objects...") + pool.clearOldest() +} + +pool.borrowObject(key) + } + + /** Returns borrowed object to the pool. */ + def returnObject(connector: V): Unit = { +pool.returnObject(createKey(connector), connector) + } + + /** Invalidates (destroy) borrowed object to the pool. */ + def invalidateObject(connector: V): Unit = { +pool.invalidateObject(createKey(connector), connector) + } + + /** Invalidates all idle values for the key */ + def invalidateKey(key: K): Unit = { +pool.clear(key) + } + + /** + * Closes the keyed object pool. Once the pool is closed, + * borrowObject will fail with [[IllegalStateException]], but returnObject and invalidateObject + * will continue to work, with returned objects destroyed on return. + * + * Also destroys idle instances in the pool. + */ + def close(): Unit = { +pool.close() + } + + def reset(): Unit = { +// this is the best-effort of clearing up. otherwise we should close the pool and create again +// but we don't want to make it "var" only because of tests. +pool.clear() + } + + def numIdle: Int = pool.getNumIdle + + def numIdle(key: K): Int = pool.getNumIdle(key) + + def numActive: Int = pool.getNumA
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326845097 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaProducerPool.scala ## @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} + +import scala.collection.JavaConverters._ + +import org.apache.commons.pool2.PooledObject + +import org.apache.spark.SparkConf +import org.apache.spark.sql.kafka010.InternalKafkaProducerPool.CacheKey + +private[kafka010] class InternalKafkaProducerPool( +objectFactory: ProducerObjectFactory, +poolConfig: ProducerPoolConfig) + extends InternalKafkaConnectorPool[CacheKey, CachedKafkaProducer]( + objectFactory, + poolConfig, + new CustomSwallowedExceptionListener("producer")) { + + def this(conf: SparkConf) = { +this(new ProducerObjectFactory, new ProducerPoolConfig(conf)) + } + + protected def createKey(producer: CachedKafkaProducer): CacheKey = { +producer.kafkaParams.asScala.toSeq.sortBy(x => x._1) Review comment: Let's use `InternalKafkaProducerPool.toCacheKey` to unify the way to create CacheKey. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326550365 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaProducer.scala ## @@ -18,111 +18,97 @@ package org.apache.spark.sql.kafka010 import java.{util => ju} -import java.util.concurrent.{ConcurrentMap, ExecutionException, TimeUnit} +import java.io.Closeable +import java.util.concurrent.ExecutionException -import com.google.common.cache._ -import com.google.common.util.concurrent.{ExecutionError, UncheckedExecutionException} -import org.apache.kafka.clients.producer.KafkaProducer import scala.collection.JavaConverters._ import scala.util.control.NonFatal +import com.google.common.util.concurrent.{ExecutionError, UncheckedExecutionException} Review comment: We shouldn't need these things anymore since we get rid of Guava cache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844741 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPool.scala ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} +import java.util.concurrent.ConcurrentHashMap + +import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, PooledObject, SwallowedExceptionListener} +import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig} + +import org.apache.spark.internal.Logging + +/** + * Provides object pool for objects which is grouped by a key. + * + * This class leverages [[GenericKeyedObjectPool]] internally, hence providing methods based on + * the class, and same contract applies: after using the borrowed object, you must either call + * returnObject() if the object is healthy to return to pool, or invalidateObject() if the object + * should be destroyed. + * + * The soft capacity of pool is determined by "poolConfig.capacity" config value, + * and the pool will have reasonable default value if the value is not provided. + * (The instance will do its best effort to respect soft capacity but it can exceed when there's + * a borrowing request and there's neither free space nor idle object to clear.) + * + * This class guarantees that no caller will get pooled object once the object is borrowed and + * not yet returned, hence provide thread-safety usage of non-thread-safe objects unless caller + * shares the object to multiple threads. + */ +private[kafka010] abstract class InternalKafkaConnectorPool[K, V]( +objectFactory: ObjectFactory[K, V], +poolConfig: PoolConfig[V], +swallowedExceptionListener: SwallowedExceptionListener) extends Logging { + + // the class is intended to have only soft capacity + assert(poolConfig.getMaxTotal < 0) + + private val pool = { +val internalPool = new GenericKeyedObjectPool[K, V](objectFactory, poolConfig) +internalPool.setSwallowedExceptionListener(swallowedExceptionListener) +internalPool + } + + /** + * Borrows object from the pool. If there's no idle object for the key, + * the pool will create the object. + * + * If the pool doesn't have idle object for the key and also exceeds the soft capacity, + * pool will try to clear some of idle objects. + * + * Borrowed object must be returned by either calling returnObject or invalidateObject, otherwise + * the object will be kept in pool as active object. + */ + def borrowObject(key: K, kafkaParams: ju.Map[String, Object]): V = { +updateKafkaParamForKey(key, kafkaParams) + +if (size >= poolConfig.softMaxSize) { + logWarning("Pool exceeds its soft max size, cleaning up idle objects...") + pool.clearOldest() +} + +pool.borrowObject(key) + } + + /** Returns borrowed object to the pool. */ + def returnObject(connector: V): Unit = { +pool.returnObject(createKey(connector), connector) + } + + /** Invalidates (destroy) borrowed object to the pool. */ + def invalidateObject(connector: V): Unit = { +pool.invalidateObject(createKey(connector), connector) + } + + /** Invalidates all idle values for the key */ + def invalidateKey(key: K): Unit = { +pool.clear(key) + } + + /** + * Closes the keyed object pool. Once the pool is closed, + * borrowObject will fail with [[IllegalStateException]], but returnObject and invalidateObject + * will continue to work, with returned objects destroyed on return. + * + * Also destroys idle instances in the pool. + */ + def close(): Unit = { +pool.close() + } + + def reset(): Unit = { +// this is the best-effort of clearing up. otherwise we should close the pool and create again +// but we don't want to make it "var" only because of tests. +pool.clear() + } + + def numIdle: Int = pool.getNumIdle + + def numIdle(key: K): Int = pool.getNumIdle(key) + + def numActive: Int = pool.getNumA
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844932 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPool.scala ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} +import java.util.concurrent.ConcurrentHashMap + +import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, PooledObject, SwallowedExceptionListener} +import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig} + +import org.apache.spark.internal.Logging + +/** + * Provides object pool for objects which is grouped by a key. + * + * This class leverages [[GenericKeyedObjectPool]] internally, hence providing methods based on + * the class, and same contract applies: after using the borrowed object, you must either call + * returnObject() if the object is healthy to return to pool, or invalidateObject() if the object + * should be destroyed. + * + * The soft capacity of pool is determined by "poolConfig.capacity" config value, + * and the pool will have reasonable default value if the value is not provided. + * (The instance will do its best effort to respect soft capacity but it can exceed when there's + * a borrowing request and there's neither free space nor idle object to clear.) + * + * This class guarantees that no caller will get pooled object once the object is borrowed and + * not yet returned, hence provide thread-safety usage of non-thread-safe objects unless caller + * shares the object to multiple threads. + */ +private[kafka010] abstract class InternalKafkaConnectorPool[K, V]( +objectFactory: ObjectFactory[K, V], +poolConfig: PoolConfig[V], +swallowedExceptionListener: SwallowedExceptionListener) extends Logging { + + // the class is intended to have only soft capacity + assert(poolConfig.getMaxTotal < 0) + + private val pool = { +val internalPool = new GenericKeyedObjectPool[K, V](objectFactory, poolConfig) +internalPool.setSwallowedExceptionListener(swallowedExceptionListener) +internalPool + } + + /** + * Borrows object from the pool. If there's no idle object for the key, + * the pool will create the object. + * + * If the pool doesn't have idle object for the key and also exceeds the soft capacity, + * pool will try to clear some of idle objects. + * + * Borrowed object must be returned by either calling returnObject or invalidateObject, otherwise + * the object will be kept in pool as active object. + */ + def borrowObject(key: K, kafkaParams: ju.Map[String, Object]): V = { +updateKafkaParamForKey(key, kafkaParams) + +if (size >= poolConfig.softMaxSize) { + logWarning("Pool exceeds its soft max size, cleaning up idle objects...") + pool.clearOldest() +} + +pool.borrowObject(key) + } + + /** Returns borrowed object to the pool. */ + def returnObject(connector: V): Unit = { +pool.returnObject(createKey(connector), connector) + } + + /** Invalidates (destroy) borrowed object to the pool. */ + def invalidateObject(connector: V): Unit = { +pool.invalidateObject(createKey(connector), connector) + } + + /** Invalidates all idle values for the key */ + def invalidateKey(key: K): Unit = { +pool.clear(key) + } + + /** + * Closes the keyed object pool. Once the pool is closed, + * borrowObject will fail with [[IllegalStateException]], but returnObject and invalidateObject + * will continue to work, with returned objects destroyed on return. + * + * Also destroys idle instances in the pool. + */ + def close(): Unit = { +pool.close() + } + + def reset(): Unit = { +// this is the best-effort of clearing up. otherwise we should close the pool and create again +// but we don't want to make it "var" only because of tests. +pool.clear() + } + + def numIdle: Int = pool.getNumIdle + + def numIdle(key: K): Int = pool.getNumIdle(key) + + def numActive: Int = pool.getNumA
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844638 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPool.scala ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} +import java.util.concurrent.ConcurrentHashMap + +import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, PooledObject, SwallowedExceptionListener} +import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig} + +import org.apache.spark.internal.Logging + +/** + * Provides object pool for objects which is grouped by a key. + * + * This class leverages [[GenericKeyedObjectPool]] internally, hence providing methods based on + * the class, and same contract applies: after using the borrowed object, you must either call + * returnObject() if the object is healthy to return to pool, or invalidateObject() if the object + * should be destroyed. + * + * The soft capacity of pool is determined by "poolConfig.capacity" config value, + * and the pool will have reasonable default value if the value is not provided. + * (The instance will do its best effort to respect soft capacity but it can exceed when there's + * a borrowing request and there's neither free space nor idle object to clear.) + * + * This class guarantees that no caller will get pooled object once the object is borrowed and + * not yet returned, hence provide thread-safety usage of non-thread-safe objects unless caller + * shares the object to multiple threads. + */ +private[kafka010] abstract class InternalKafkaConnectorPool[K, V]( Review comment: For other reviewers: this is pretty same as previous InternalKafkaConsumerPool, except 1. It brings some abstract method to deal with extracting key from object. 2. It replaces "consumer" related words in javadoc/code to common one to apply this to both consumer and producer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844478 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConnectorPool.scala ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.kafka010 + +import java.{util => ju} +import java.util.concurrent.ConcurrentHashMap + +import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, PooledObject, SwallowedExceptionListener} +import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig} + +import org.apache.spark.internal.Logging + +/** + * Provides object pool for objects which is grouped by a key. + * + * This class leverages [[GenericKeyedObjectPool]] internally, hence providing methods based on + * the class, and same contract applies: after using the borrowed object, you must either call + * returnObject() if the object is healthy to return to pool, or invalidateObject() if the object + * should be destroyed. + * + * The soft capacity of pool is determined by "poolConfig.capacity" config value, + * and the pool will have reasonable default value if the value is not provided. + * (The instance will do its best effort to respect soft capacity but it can exceed when there's + * a borrowing request and there's neither free space nor idle object to clear.) + * + * This class guarantees that no caller will get pooled object once the object is borrowed and + * not yet returned, hence provide thread-safety usage of non-thread-safe objects unless caller + * shares the object to multiple threads. + */ +private[kafka010] abstract class InternalKafkaConnectorPool[K, V]( +objectFactory: ObjectFactory[K, V], +poolConfig: PoolConfig[V], +swallowedExceptionListener: SwallowedExceptionListener) extends Logging { + + // the class is intended to have only soft capacity + assert(poolConfig.getMaxTotal < 0) + + private val pool = { +val internalPool = new GenericKeyedObjectPool[K, V](objectFactory, poolConfig) +internalPool.setSwallowedExceptionListener(swallowedExceptionListener) +internalPool + } + + /** + * Borrows object from the pool. If there's no idle object for the key, + * the pool will create the object. + * + * If the pool doesn't have idle object for the key and also exceeds the soft capacity, + * pool will try to clear some of idle objects. + * + * Borrowed object must be returned by either calling returnObject or invalidateObject, otherwise + * the object will be kept in pool as active object. + */ + def borrowObject(key: K, kafkaParams: ju.Map[String, Object]): V = { +updateKafkaParamForKey(key, kafkaParams) + +if (size >= poolConfig.softMaxSize) { + logWarning("Pool exceeds its soft max size, cleaning up idle objects...") + pool.clearOldest() +} + +pool.borrowObject(key) + } + + /** Returns borrowed object to the pool. */ + def returnObject(connector: V): Unit = { +pool.returnObject(createKey(connector), connector) + } + + /** Invalidates (destroy) borrowed object to the pool. */ + def invalidateObject(connector: V): Unit = { +pool.invalidateObject(createKey(connector), connector) + } + + /** Invalidates all idle values for the key */ + def invalidateKey(key: K): Unit = { +pool.clear(key) + } + + /** + * Closes the keyed object pool. Once the pool is closed, + * borrowObject will fail with [[IllegalStateException]], but returnObject and invalidateObject + * will continue to work, with returned objects destroyed on return. + * + * Also destroys idle instances in the pool. + */ + def close(): Unit = { +pool.close() + } + + def reset(): Unit = { +// this is the best-effort of clearing up. otherwise we should close the pool and create again +// but we don't want to make it "var" only because of tests. +pool.clear() + } + + def numIdle: Int = pool.getNumIdle + + def numIdle(key: K): Int = pool.getNumIdle(key) + + def numActive: Int = pool.getNumA
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326845411 ## File path: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDataConsumerSuite.scala ## @@ -148,7 +148,7 @@ class KafkaDataConsumerSuite extends SharedSparkSession with PrivateMethodTester @volatile var error: Throwable = null -def consume(i: Int): Unit = { +def consume(): Unit = { Review comment: For other reviewers: this removes unused parameter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326844184 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaProducer.scala ## @@ -18,111 +18,97 @@ package org.apache.spark.sql.kafka010 import java.{util => ju} -import java.util.concurrent.{ConcurrentMap, ExecutionException, TimeUnit} +import java.io.Closeable +import java.util.concurrent.ExecutionException -import com.google.common.cache._ -import com.google.common.util.concurrent.{ExecutionError, UncheckedExecutionException} -import org.apache.kafka.clients.producer.KafkaProducer import scala.collection.JavaConverters._ import scala.util.control.NonFatal +import com.google.common.util.concurrent.{ExecutionError, UncheckedExecutionException} +import org.apache.kafka.clients.producer.{Callback, KafkaProducer, ProducerRecord} + import org.apache.spark.SparkEnv import org.apache.spark.internal.Logging import org.apache.spark.kafka010.{KafkaConfigUpdater, KafkaRedactionUtil} +import org.apache.spark.sql.kafka010.InternalKafkaProducerPool._ +import org.apache.spark.util.ShutdownHookManager -private[kafka010] object CachedKafkaProducer extends Logging { +private[kafka010] class CachedKafkaProducer(val kafkaParams: ju.Map[String, Object]) + extends Closeable with Logging { private type Producer = KafkaProducer[Array[Byte], Array[Byte]] - private val defaultCacheExpireTimeout = TimeUnit.MINUTES.toMillis(10) - - private lazy val cacheExpireTimeout: Long = Option(SparkEnv.get) -.map(_.conf.get(PRODUCER_CACHE_TIMEOUT)) -.getOrElse(defaultCacheExpireTimeout) + private val producer = createProducer() - private val cacheLoader = new CacheLoader[Seq[(String, Object)], Producer] { -override def load(config: Seq[(String, Object)]): Producer = { - createKafkaProducer(config) + private def createProducer(): Producer = { +val producer: Producer = new Producer(kafkaParams) +if (log.isDebugEnabled()) { + val redactedParamsSeq = KafkaRedactionUtil.redactParams(toCacheKey(kafkaParams)) + logDebug(s"Created a new instance of kafka producer for $redactedParamsSeq.") } +producer } - private val removalListener = new RemovalListener[Seq[(String, Object)], Producer]() { -override def onRemoval( -notification: RemovalNotification[Seq[(String, Object)], Producer]): Unit = { - val paramsSeq: Seq[(String, Object)] = notification.getKey - val producer: Producer = notification.getValue - if (log.isDebugEnabled()) { -val redactedParamsSeq = KafkaRedactionUtil.redactParams(paramsSeq) -logDebug(s"Evicting kafka producer $producer params: $redactedParamsSeq, " + - s"due to ${notification.getCause}") + override def close(): Unit = { +try { + if (log.isInfoEnabled()) { +val redactedParamsSeq = KafkaRedactionUtil.redactParams(toCacheKey(kafkaParams)) +logInfo(s"Closing the KafkaProducer with params: ${redactedParamsSeq.mkString("\n")}.") } - close(paramsSeq, producer) + producer.close() +} catch { + case NonFatal(e) => logWarning("Error while closing kafka producer.", e) } } - private lazy val guavaCache: LoadingCache[Seq[(String, Object)], Producer] = -CacheBuilder.newBuilder().expireAfterAccess(cacheExpireTimeout, TimeUnit.MILLISECONDS) - .removalListener(removalListener) - .build[Seq[(String, Object)], Producer](cacheLoader) + def send(record: ProducerRecord[Array[Byte], Array[Byte]], callback: Callback): Unit = { +producer.send(record, callback) + } - private def createKafkaProducer(paramsSeq: Seq[(String, Object)]): Producer = { -val kafkaProducer: Producer = new Producer(paramsSeq.toMap.asJava) -if (log.isDebugEnabled()) { - val redactedParamsSeq = KafkaRedactionUtil.redactParams(paramsSeq) - logDebug(s"Created a new instance of KafkaProducer for $redactedParamsSeq.") + def flush(): Unit = { +producer.flush() + } +} + +private[kafka010] object CachedKafkaProducer extends Logging { + + private val sparkConf = SparkEnv.get.conf + private val producerPool = new InternalKafkaProducerPool(sparkConf) + + ShutdownHookManager.addShutdownHook { () => +try { + producerPool.close() +} catch { + case e: Throwable => +logWarning("Ignoring exception while shutting down pool from shutdown hook", e) } -kafkaProducer } /** * Get a cached KafkaProducer for a given configuration. If matching KafkaProducer doesn't * exist, a new KafkaProducer will be created. KafkaProducer is thread safe, it is best to keep * one instance per specified kafkaParams. */ - private[kafka010] def getOrCreate(kafkaParams: ju.Map[String, Object]): Producer = { -val updatedKafkaProducerConfiguration = + def acquir
[GitHub] [spark] HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer
HeartSaVioR commented on a change in pull request #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#discussion_r326845141 ## File path: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala ## @@ -619,6 +619,10 @@ private[kafka010] object KafkaDataConsumer extends Logging { new KafkaDataConsumer(topicPartition, kafkaParams, consumerPool, fetchedDataPool) } + private[kafka010] def clear(): Unit = { +consumerPool.reset() Review comment: Let's call `fetchedDataPool.reset()` as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions
AmplabJenkins removed a comment on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions URL: https://github.com/apache/spark/pull/25830#issuecomment-533763719 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions
AmplabJenkins removed a comment on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions URL: https://github.com/apache/spark/pull/25830#issuecomment-533763721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25874: [SPARK-28772][BUILD][MLLIB] Update breeze to 1.0
dongjoon-hyun closed pull request #25874: [SPARK-28772][BUILD][MLLIB] Update breeze to 1.0 URL: https://github.com/apache/spark/pull/25874 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions
AmplabJenkins commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions URL: https://github.com/apache/spark/pull/25830#issuecomment-533763719 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions
AmplabJenkins commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions URL: https://github.com/apache/spark/pull/25830#issuecomment-533763721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25874: [SPARK-28772][BUILD][MLLIB] Update breeze to 1.0
dongjoon-hyun commented on issue #25874: [SPARK-28772][BUILD][MLLIB] Update breeze to 1.0 URL: https://github.com/apache/spark/pull/25874#issuecomment-533763726 Merged to master. Thank you, @srowen . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25865: [SPARK-29187][SQL] Return null from `date_part()` for the null `field`
dongjoon-hyun closed pull request #25865: [SPARK-29187][SQL] Return null from `date_part()` for the null `field` URL: https://github.com/apache/spark/pull/25865 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25771: [SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2
imback82 commented on a change in pull request #25771: [SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2 URL: https://github.com/apache/spark/pull/25771#discussion_r326845256 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceResolution.scala ## @@ -210,6 +197,17 @@ case class DataSourceResolution( } ShowTablesCommand(Some(namespace.quoted), pattern) } + +case UseCatalogAndNamespaceStatement(catalogName, namespace) => + catalogName match { +case Some(c) => + validateCatalog(c) + UseCatalogAndNamespace(catalogManager, Some(c), namespace) +case None => + require(namespace.nonEmpty) + val CurrentCatalogAndNamespace(catalog, ns) = namespace.get + UseCatalogAndNamespace(catalogManager, Some(catalog.name()), Some(ns)) Review comment: Actually, this may not be correct if the catalog is `V2SessionCatalog`. The V1 behavior is that it calls `SessionCatalog.setCurrentDatabase(databaseName)`. So if we don't fallback to V1, any code that access `SessionCatalog.getCurrentDatabase` can be broken. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions
SparkQA removed a comment on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions URL: https://github.com/apache/spark/pull/25830#issuecomment-533743421 **[Test build #111096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111096/testReport)** for PR 25830 at commit [`4c00a2b`](https://github.com/apache/spark/commit/4c00a2b5386f318fbbded9e7f7e1daee7782758b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25865: [SPARK-29187][SQL] Return null from `date_part()` for the null `field`
dongjoon-hyun commented on issue #25865: [SPARK-29187][SQL] Return null from `date_part()` for the null `field` URL: https://github.com/apache/spark/pull/25865#issuecomment-533763676 Also, thank you for review, @adrian-wang . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions
SparkQA commented on issue #25830: [SPARK-29140][SQL] Handle parameters having "array" of javaType properly in splitAggregateExpressions URL: https://github.com/apache/spark/pull/25830#issuecomment-533763611 **[Test build #111096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111096/testReport)** for PR 25830 at commit [`4c00a2b`](https://github.com/apache/spark/commit/4c00a2b5386f318fbbded9e7f7e1daee7782758b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25807: [SPARK-25153][SQL] Improve error messages for columns with dots/periods
dongjoon-hyun commented on issue #25807: [SPARK-25153][SQL] Improve error messages for columns with dots/periods URL: https://github.com/apache/spark/pull/25807#issuecomment-533763283 BTW, thank you for making a PR, @jeff303 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org