[GitHub] [spark] dongjoon-hyun commented on issue #25794: [SPARK-26989][CORE][TEST][2.4] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values
dongjoon-hyun commented on issue #25794: [SPARK-26989][CORE][TEST][2.4] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values URL: https://github.com/apache/spark/pull/25794#issuecomment-531537318 Thank you so much for making this backporting PR, @HeartSaVioR . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25794: [SPARK-26989][CORE][TEST][2.4] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values
SparkQA commented on issue #25794: [SPARK-26989][CORE][TEST][2.4] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values URL: https://github.com/apache/spark/pull/25794#issuecomment-531537306 **[Test build #110609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110609/testReport)** for PR 25794 at commit [`ed50055`](https://github.com/apache/spark/commit/ed5005558d10c19e997be04166639fa9fe45c264). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR opened a new pull request #25794: [SPARK-26989][CORE][TEST] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values
HeartSaVioR opened a new pull request #25794: [SPARK-26989][CORE][TEST] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values URL: https://github.com/apache/spark/pull/25794 ### What changes were proposed in this pull request? This patch ensures accessing recorded values in listener is always after letting listeners fully process all events. To ensure this, this patch adds new class to hide these values and access with methods which will ensure above condition. Without this guard, two threads are running concurrently - 1) listeners process thread 2) test main thread - and race condition would occur. That's why we also see very odd thing, error message saying condition is met but test failed: ``` - Barrier task failures from the same stage attempt don't trigger multiple stage retries *** FAILED *** ArrayBuffer(0) did not equal List(0) (DAGSchedulerSuite.scala:2656) ``` which means verification failed, and condition is met just before constructing error message. The guard is properly placed in many spots, but missed in some places. This patch enforces that it can't be missed. ### Why are the changes needed? UT fails intermittently and this patch will address the flakyness. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Modified UT. Also made the flaky tests artificially failing via applying 50ms of sleep on each onXXX method.  I found 3 methods being failed. (They've marked as X. Just ignore ! as they failed on waiting listener in given timeout and these tests don't deal with these recorded values - it uses other timeout value 1000ms than 1ms for this listener so affected via side-effect.) When I applied same in this patch all tests marked as X passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
dongjoon-hyun edited a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531537068 Do you think we need to tigger this PR multiple times, @wangyum ? Among the last three runs, two runs hit Hive test failures. I'm just wondering if those two Hive failures are relevant or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
dongjoon-hyun commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531537068 Ur, do we think we need to tigger this PR multiple times, @wangyum ? Among the last three runs, two runs hit Hive test failures. I'm just wondering if those two Hive failures are relevant or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531536754 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531536756 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110605/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531536756 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110605/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531536754 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
SparkQA removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531531261 **[Test build #110605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110605/testReport)** for PR 25690 at commit [`36c394c`](https://github.com/apache/spark/commit/36c394ce08e6cac1e32176c684eac0c9d1615831). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
SparkQA commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531536668 **[Test build #110605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110605/testReport)** for PR 25690 at commit [`36c394c`](https://github.com/apache/spark/commit/36c394ce08e6cac1e32176c684eac0c9d1615831). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25786: [SPARK-29079][INFRA] Enable GitHub Action on PR
dongjoon-hyun commented on issue #25786: [SPARK-29079][INFRA] Enable GitHub Action on PR URL: https://github.com/apache/spark/pull/25786#issuecomment-531536069 It seems working correctly on `master`. I'll logically backport this to `branch-2.4`, too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
HeartSaVioR commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531535938 cc. @zsxwing @srowen as reviewers with commit privileges of #24173 , and @dongjinleekr as author of #24173 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
HeartSaVioR commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531535795 Actually the other intention was let test fail when "includeHeaders" is set to "true", as we suspected from https://github.com/apache/spark/pull/22282#issuecomment-463008868, but turns out it doesn't break even the option is set to true. It's because of the mechanism how `dropDuplicate` works - row is stored as key, and getting value via key doesn't leverage type while comparing as both hashCode and equals of row don't use type. Luckily it doesn't let query crash, but it would be ideal to do type check and avoid this case (possible chance to crash in runtime) - SPARK-27237 (#24173) has been proposed to address this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24088: [SPARK-27122][core] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
dongjoon-hyun commented on issue #24088: [SPARK-27122][core] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/24088#issuecomment-531535495 I made a backport PR to `branch-2.4`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
dongjoon-hyun commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793#issuecomment-531535462 cc @ajithme , @srowen , @cloud-fan , @vanzin , @dbtsai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
SparkQA commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793#issuecomment-531535441 **[Test build #110608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110608/testReport)** for PR 25793 at commit [`79ae1ba`](https://github.com/apache/spark/commit/79ae1bab26f59542893ff1ae9480711ddc119ee7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
AmplabJenkins removed a comment on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793#issuecomment-531535345 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
AmplabJenkins removed a comment on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793#issuecomment-531535347 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15585/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
AmplabJenkins commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793#issuecomment-531535347 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15585/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
AmplabJenkins commented on issue #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793#issuecomment-531535345 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
dongjoon-hyun opened a new pull request #25793: [SPARK-27122][CORE][2.4] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI URL: https://github.com/apache/spark/pull/25793 ### What changes were proposed in this pull request? This is a backport of #24088. ### Why are the changes needed? When we run YarnSchedulerBackendSuite, the class path seems to be made from the classes folder(resource-managers/yarn/target/scala-2.12/classes) instead of jar (resource-managers/yarn/target/spark-yarn_2.12-3.0.0-SNAPSHOT.jar) . ui.getHandlers is in spark-core and its loaded from spark-core.jar which is shaded and hence refers to org.spark_project.jetty.servlet.ServletContextHandler Here in org.apache.spark.scheduler.cluster.YarnSchedulerBackend, as its not shaded, it expects org.eclipse.jetty.servlet.ServletContextHandler Refer discussion https://issues.apache.org/jira/browse/SPARK-27122?focusedCommentId=16792318=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16792318 Hence as a fix, org.apache.spark.ui.WebUI must only return a wrapper class instance or references so that Jetty classes can be avoided in getters which are accessed outside spark-core ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins with the existing tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
SparkQA commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531533754 **[Test build #110607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110607/testReport)** for PR 25791 at commit [`bed11a8`](https://github.com/apache/spark/commit/bed11a85e572bfc59803d547e6dab493ca4fa659). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size`
AmplabJenkins commented on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size` URL: https://github.com/apache/spark/pull/25689#issuecomment-531533650 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size`
AmplabJenkins removed a comment on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size` URL: https://github.com/apache/spark/pull/25689#issuecomment-531533650 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531533638 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size`
AmplabJenkins commented on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size` URL: https://github.com/apache/spark/pull/25689#issuecomment-531533651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15584/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size`
AmplabJenkins removed a comment on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size` URL: https://github.com/apache/spark/pull/25689#issuecomment-531533651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15584/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531533638 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531533641 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15583/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531533641 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15583/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size`
SparkQA commented on issue #25689: [SPARK-28972][DOCS] Describe unit for `spark.memory.offHeap.size` URL: https://github.com/apache/spark/pull/25689#issuecomment-531533469 **[Test build #110606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110606/testReport)** for PR 25689 at commit [`13bdfaa`](https://github.com/apache/spark/commit/13bdfaa316a83c5520ffd4a7213ded417510df3e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
dongjoon-hyun commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531533542 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531532979 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110603/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
SparkQA removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529258 **[Test build #110603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110603/testReport)** for PR 25791 at commit [`bed11a8`](https://github.com/apache/spark/commit/bed11a85e572bfc59803d547e6dab493ca4fa659). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531532978 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531532978 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
SparkQA commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531532950 **[Test build #110603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110603/testReport)** for PR 25791 at commit [`bed11a8`](https://github.com/apache/spark/commit/bed11a85e572bfc59803d547e6dab493ca4fa659). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531532979 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110603/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25789: [SPARK-28927][ML] Show warning when input data to ALS is indeterminate
viirya commented on a change in pull request #25789: [SPARK-28927][ML] Show
warning when input data to ALS is indeterminate
URL: https://github.com/apache/spark/pull/25789#discussion_r32761
##
File path: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
##
@@ -920,6 +924,14 @@ object ALS extends DefaultParamsReadable[ALS] with
Logging {
require(intermediateRDDStorageLevel != StorageLevel.NONE,
"ALS is not designed to run without persisting intermediate RDDs.")
+// Indeterminate rating RDD causes inconsistent in/out blocks in case of
rerun.
+// It can cause runtime error when matching in/out user/item blocks.
+if (ratings.outputDeterministicLevel == DeterministicLevel.INDETERMINATE) {
Review comment:
Yes, it is possible. It is just hard to tell if it is the case during ALS
fitting time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #25785: [SPARK-27812][CORE] Explicit System.exit after job's main
srowen commented on issue #25785: [SPARK-27812][CORE] Explicit System.exit after job's main URL: https://github.com/apache/spark/pull/25785#issuecomment-531531304 Applications must call `stop()`. More things don't stop if an app doesn't, when it intends to finish. If things work after calling `stop()` I think it's already working as intended. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
SparkQA commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531531261 **[Test build #110605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110605/testReport)** for PR 25690 at commit [`36c394c`](https://github.com/apache/spark/commit/36c394ce08e6cac1e32176c684eac0c9d1615831). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531531204 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15582/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins removed a comment on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531531203 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531531204 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15582/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
AmplabJenkins commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531531203 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
AmplabJenkins removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531530951 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110604/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
AmplabJenkins removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531530948 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
SparkQA removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531529850 **[Test build #110604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110604/testReport)** for PR 25792 at commit [`a3b5824`](https://github.com/apache/spark/commit/a3b5824eca2b972eab3bd8cf50bb898a17a4a299). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
AmplabJenkins commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531530951 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110604/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
AmplabJenkins commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531530948 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file
wangyum commented on issue #25690: [SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file URL: https://github.com/apache/spark/pull/25690#issuecomment-531530905 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
SparkQA commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531530915 **[Test build #110604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110604/testReport)** for PR 25792 at commit [`a3b5824`](https://github.com/apache/spark/commit/a3b5824eca2b972eab3bd8cf50bb898a17a4a299). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
HyukjinKwon commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531530073 Thanks all for taking care of this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
HyukjinKwon removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531521315 @dongjoon-hyun, I think everybody knows and does it. It's been public holidays in KST and I guess it's weekends - so the action was a bit delayed. Commit / revert thing can happen and I don't think there's particularly bad pattern here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
dongjoon-hyun removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531506514 To committers. Since 3.0.0-preview is coming, please take a look the Jenkins status after your merging. We can notice this easily because this causes consecutive outages across over all Jenkins combinations from Sep 11, 2019 9:50PM to Sep 14, 2019 12:16AM without fixing the root causes. Also, many irrelevant PRs suffered during that period due to the flakiness. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
dongjoon-hyun removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531524525 Of course, I knew that it was due to a big holiday for Lunar calendar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
AmplabJenkins removed a comment on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531529682 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders
SparkQA commented on issue #25792: [SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531529850 **[Test build #110604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110604/testReport)** for PR 25792 at commit [`a3b5824`](https://github.com/apache/spark/commit/a3b5824eca2b972eab3bd8cf50bb898a17a4a299). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
dongjoon-hyun commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529800 cc @ajithme and @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25792: [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders
AmplabJenkins commented on issue #25792: [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531529682 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25792: [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders
HeartSaVioR commented on issue #25792: [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792#issuecomment-531529644 This patch is to address my own comment which PR was merged without addressing this. https://github.com/apache/spark/pull/22282#issuecomment-476883917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR opened a new pull request #25792: [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders
HeartSaVioR opened a new pull request #25792: [SPARK-23539][SS][FOLLOWUP] Add UT to ensure existing query doesn't break with default conf of includeHeaders URL: https://github.com/apache/spark/pull/25792 ### What changes were proposed in this pull request? This patch adds new UT to ensure existing query (before Spark 3.0.0) with checkpoint doesn't break with default configuration of "includeHeaders". This patch also modifies existing test which checks type of columns to also check headers column as well. ### Why are the changes needed? The patch adds missing tests which guarantees backward compatibility of the change of SPARK-23539. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? UT passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531529516 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110602/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531529515 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531529516 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110602/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531529515 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529433 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15581/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529433 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15581/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins removed a comment on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529431 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
AmplabJenkins commented on issue #25791: [SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529431 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
SparkQA removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531519734 **[Test build #110602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110602/testReport)** for PR 25717 at commit [`71e0239`](https://github.com/apache/spark/commit/71e02390d67d9671c4e3dcac586b9788d0ecd050). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
SparkQA commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531529441 **[Test build #110602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110602/testReport)** for PR 25717 at commit [`71e0239`](https://github.com/apache/spark/commit/71e02390d67d9671c4e3dcac586b9788d0ecd050). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25791: [SPARK-29087][SS] Use DelegatingServletContextHandler to avoid CCE
SparkQA commented on issue #25791: [SPARK-29087][SS] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791#issuecomment-531529258 **[Test build #110603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110603/testReport)** for PR 25791 at commit [`bed11a8`](https://github.com/apache/spark/commit/bed11a85e572bfc59803d547e6dab493ca4fa659). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25791: [SPARK-29087][SS] Use DelegatingServletContextHandler to avoid CCE
dongjoon-hyun opened a new pull request #25791: [SPARK-29087][SS] Use DelegatingServletContextHandler to avoid CCE URL: https://github.com/apache/spark/pull/25791 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #25789: [SPARK-28927][ML] Show warning when input data to ALS is indeterminate
felixcheung commented on a change in pull request #25789: [SPARK-28927][ML]
Show warning when input data to ALS is indeterminate
URL: https://github.com/apache/spark/pull/25789#discussion_r324443661
##
File path: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
##
@@ -920,6 +924,14 @@ object ALS extends DefaultParamsReadable[ALS] with
Logging {
require(intermediateRDDStorageLevel != StorageLevel.NONE,
"ALS is not designed to run without persisting intermediate RDDs.")
+// Indeterminate rating RDD causes inconsistent in/out blocks in case of
rerun.
+// It can cause runtime error when matching in/out user/item blocks.
+if (ratings.outputDeterministicLevel == DeterministicLevel.INDETERMINATE) {
Review comment:
if I understanding this correctly, mismatch -> failure is only one possible
outcome? it could also end up matching the wrong user/item before the index is
wrong? that seems more subtle and much harder to detect
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively
AmplabJenkins removed a comment on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively URL: https://github.com/apache/spark/pull/25778#issuecomment-531526024 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110601/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively
AmplabJenkins removed a comment on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively URL: https://github.com/apache/spark/pull/25778#issuecomment-531526023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively
AmplabJenkins commented on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively URL: https://github.com/apache/spark/pull/25778#issuecomment-531526024 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110601/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively
AmplabJenkins commented on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively URL: https://github.com/apache/spark/pull/25778#issuecomment-531526023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively
SparkQA removed a comment on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively URL: https://github.com/apache/spark/pull/25778#issuecomment-531519347 **[Test build #110601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110601/testReport)** for PR 25778 at commit [`8075892`](https://github.com/apache/spark/commit/8075892cd291600a6a80de58962f73d3c1093252). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively
SparkQA commented on issue #25778: [SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively URL: https://github.com/apache/spark/pull/25778#issuecomment-531525933 **[Test build #110601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110601/testReport)** for PR 25778 at commit [`8075892`](https://github.com/apache/spark/commit/8075892cd291600a6a80de58962f73d3c1093252). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] igorcalabria commented on issue #25785: [SPARK-27812][CORE] Explicit System.exit after job's main
igorcalabria commented on issue #25785: [SPARK-27812][CORE] Explicit System.exit after job's main URL: https://github.com/apache/spark/pull/25785#issuecomment-531524626 > I see, can that thread be a daemon? If System.exit is viable (i.e. immediately stopping daemon threads) then it should be. But if not, then yeah such a thread needs to be shut down cleanly somehow during the shutdown process. This could be a shutdown hook. I don't think there's an option to create a daemon thread in this case. This was already discussed on https://github.com/square/okhttp/issues/3339 I'm sorry, but I didn't understand what you meant about the viability of `System.exit`. There's already a shutdown logic for the kubernetes client https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L115 but this only called if the shutdownHook runs or if the user calls `SparkSession#stop()`. Calling system.exit is just a way to ensure that the hooks run after the user's main code. Everything is still cleaned up nicely since the System.exit does not bypass the registered hooks. > Hi, @igorcalabria . > According to the test result, this PR seems to break `Spark Thrift Server` module at least UT level. The following is one example failure. To be considered as mergeable, this PR should pass all UTs at least. Could you focus on fixing that module? > > ``` > $ build/sbt -Phive-thriftserver "hive-thriftserver/test-only *.SingleSessionSuite" > ... > [info] Tests: succeeded 0, failed 3, canceled 0, ignored 0, pending 0 > [info] *** 3 TESTS FAILED *** > [error] Failed: Total 3, Failed 3, Errors 0, Passed 0 > [error] Failed tests: > [error]org.apache.spark.sql.hive.thriftserver.SingleSessionSuite > [error] (hive-thriftserver/test:testOnly) sbt.TestsFailedException: Tests unsuccessful > [error] Total time: 258 s, completed Sep 14, 2019 4:16:09 PM > ``` @dongjoon-hyun I'll take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
dongjoon-hyun commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531524525 Of course, I knew that it was due to a big holiday for Lunar calendar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
viirya commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531524227 > btw, this pr doesn't include end-to-end tests and do you think the queries in the existing tests are enough for the end-to-end tests of this pr? I'm not sure about how much the existing tests include structurally equivalent exprs though... Regarding end-to-end tests, I am not sure what test we need. If there are end-to-end tests of current sub-expresssion elimination, I think I can write tests based on that. > Also, no negative performance impacts on the existing queries, e.g., TPCDS? I have not run a TPCDS benchmark with this. Just `TPCDSQueryBenchmark` is enough? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
maropu edited a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531522434 btw, this pr doesn't include end-to-end tests and do you think the queries in the existing tests are enough for the end-to-end tests of this pr? I'm not sure about how much the existing tests include structurally equivalent exprs though... Also, no negative performance impacts on the existing queries, e.g., TPCDS? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
maropu commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531522434 btw, this pr doesn't include end-to-end tests and do you think the queries in the existing tests are enough for the end-to-end tests of this pr? I'm not sure about how much the existing tests include structurally equivalent exprs though... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
maropu commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531521923 @cloud-fan @rednaxelafx @mgaido91 @kiszk Anyone can check this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
maropu commented on a change in pull request #25717: [SPARK-29013][SQL]
Structurally equivalent subexpression elimination
URL: https://github.com/apache/spark/pull/25717#discussion_r324441285
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
##
@@ -40,23 +40,47 @@ class EquivalentExpressions {
override def hashCode: Int = e.semanticHash()
}
+ /**
+ * Wrapper around an Expression that provides structural semantic equality.
+ */
+ case class StructuralExpr(e: Expression) {
+def normalized(expr: Expression): Expression = {
+ expr.transformUp {
+case b: ParameterizedBoundReference =>
Review comment:
The current one looks super good to me. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
HyukjinKwon edited a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531521315 @dongjoon-hyun, I think everybody knows and does it. It's been public holidays in KST and I guess it's weekends - so the action was a bit delayed. Commit / revert thing can happen and I don't think there's particularly bad pattern here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
HyukjinKwon commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531521333 > @dongjoon-hyun, I think everybody knows and does it. It's been public holidays in KST and I guess it's weekends - so the action was a bit delayed. Commit / revert thing can happen and I don't think there's particularly bad pattern here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
HyukjinKwon removed a comment on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531521333 > @dongjoon-hyun, I think everybody knows and does it. It's been public holidays in KST and I guess it's weekends - so the action was a bit delayed. Commit / revert thing can happen and I don't think there's particularly bad pattern here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
HyukjinKwon commented on issue #25753: [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25753#issuecomment-531521315 @dongjoon-hyun, I think everybody knows and does it. It's been public holidays in KST and I guess it's weekends - so the action was a it delayed. Commit / revert thing can happen and I don't think there's particularly bad pattern here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531520200 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531520201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110598/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531520201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110598/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
AmplabJenkins commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531520200 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
SparkQA removed a comment on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531505938 **[Test build #110598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110598/testReport)** for PR 25717 at commit [`f447042`](https://github.com/apache/spark/commit/f44704222e270048ba5abf0ac1224d1182b8e31f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
SparkQA commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531520115 **[Test build #110598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110598/testReport)** for PR 25717 at commit [`f447042`](https://github.com/apache/spark/commit/f44704222e270048ba5abf0ac1224d1182b8e31f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination
SparkQA commented on issue #25717: [SPARK-29013][SQL] Structurally equivalent subexpression elimination URL: https://github.com/apache/spark/pull/25717#issuecomment-531519734 **[Test build #110602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110602/testReport)** for PR 25717 at commit [`71e0239`](https://github.com/apache/spark/commit/71e02390d67d9671c4e3dcac586b9788d0ecd050). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
