[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @vanzin @squito @srowen @jiangxb1987 @henryr Big thanks to everybody for the constructive comments, learned a lot from them. I'll take a look at further possibilities like the suggested spark-hive module. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 Merging to master. It would be nice to file a separate bug to eventually look at how to do this on the spark-hive module (or maybe it's just not worth the effort). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86094/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86094/testReport)** for PR 19893 at commit [`68d0f3b`](https://github.com/apache/spark/commit/68d0f3beac63fefba436646896d5cd82a3c4da21). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Checked and seems like unrelated. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86093/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86093/testReport)** for PR 19893 at commit [`56a41df`](https://github.com/apache/spark/commit/56a41dfece2513cd31ca10e7f4529d4b524a0814). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * ` * That trait initializes the spark session in its [[beforeAll()]] implementation before the` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86094/testReport)** for PR 19893 at commit [`68d0f3b`](https://github.com/apache/spark/commit/68d0f3beac63fefba436646896d5cd82a3c4da21). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86093/testReport)** for PR 19893 at commit [`56a41df`](https://github.com/apache/spark/commit/56a41dfece2513cd31ca10e7f4529d4b524a0814). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Thread audit disabled in hive. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86050/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86050/testReport)** for PR 19893 at commit [`9c9c6ef`](https://github.com/apache/spark/commit/9c9c6ef05e6ed6d1ad1a50591a0ed84811876bd7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 Why not disable the thread audit in the hive module? You added that functionality already, should be pretty trivial to use it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Related hive please see my comment on 11 Dec 2017. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86050/testReport)** for PR 19893 at commit [`9c9c6ef`](https://github.com/apache/spark/commit/9c9c6ef05e6ed6d1ad1a50591a0ed84811876bd7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19893 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Checked the failure but seems like unrelated. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86035/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86035/testReport)** for PR 19893 at commit [`9c9c6ef`](https://github.com/apache/spark/commit/9c9c6ef05e6ed6d1ad1a50591a0ed84811876bd7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #86035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86035/testReport)** for PR 19893 at commit [`9c9c6ef`](https://github.com/apache/spark/commit/9c9c6ef05e6ed6d1ad1a50591a0ed84811876bd7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85802/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85802/testReport)** for PR 19893 at commit [`87c4852`](https://github.com/apache/spark/commit/87c48520ef7c3236c217cdfd787e048866fcb75e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85802/testReport)** for PR 19893 at commit [`87c4852`](https://github.com/apache/spark/commit/87c48520ef7c3236c217cdfd787e048866fcb75e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85722/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)** for PR 19893 at commit [`0851ef2`](https://github.com/apache/spark/commit/0851ef2b572ed507155358e541a4fa53a8e92173). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)** for PR 19893 at commit [`0851ef2`](https://github.com/apache/spark/commit/0851ef2b572ed507155358e541a4fa53a8e92173). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85315/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85315/testReport)** for PR 19893 at commit [`f7939fa`](https://github.com/apache/spark/commit/f7939fab92a205168c9713e371d490f5c036577e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85315/testReport)** for PR 19893 at commit [`f7939fa`](https://github.com/apache/spark/commit/f7939fab92a205168c9713e371d490f5c036577e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85290/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85290/testReport)** for PR 19893 at commit [`ef00796`](https://github.com/apache/spark/commit/ef007969a576dd5f84440ce78a325d6617dc5278). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85290/testReport)** for PR 19893 at commit [`ef00796`](https://github.com/apache/spark/commit/ef007969a576dd5f84440ce78a325d6617dc5278). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19893 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 gentle ping @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @squito I mean another jira, because it needs deeper analysis and discussion. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 lgtm @jiangxb1987 are you still looking at this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 ok I just took a look at BroadcastExchangeExec, I see what you mean. It isn't *that* bad, since spark isn't continually creating more instances of those threads (you're not supposed to have more than one spark context in one jvm anyway ...). But it also means a bad test could mess up that thread pool (send it some task which never completes) and it would screw up all future tests. I'd file a jira about it, but don't think you need to worry about it for this change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 > The last suspicious big group of threads (at least for me) is broadcast-exchange.* but as I've seen this is not false positive because the threadpool never stopped. In BroadcastExchangeExec:141 a new daemon thread pool created in case of broadcast join which dies when jvm dies. I suggest to solve this separately. what do you mean about handling this separately? should it be whitelisted for now, or should there be another jira for it? It seems fine to me to set it aside for now, just want to be clear what the next steps are. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84915/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84915/testReport)** for PR 19893 at commit [`ef00796`](https://github.com/apache/spark/commit/ef007969a576dd5f84440ce78a325d6617dc5278). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 The last suspicious big group of threads (at least for me) is broadcast-exchange.* but as I've seen this is not false positive because the threadpool never stopped. In BroadcastExchangeExec:141 a new daemon thread pool created in case of broadcast join which dies when jvm dies. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84915/testReport)** for PR 19893 at commit [`ef00796`](https://github.com/apache/spark/commit/ef007969a576dd5f84440ce78a325d6617dc5278). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Seems like the new feature caught some false positives in SQL: ``` = THREAD AUDIT POST ACTION CALLED WITHOUT PRE ACTION IN SUITE o.a.s.sql.sources.DataSourceAnalysisSuite = = THREAD AUDIT POST ACTION CALLED WITHOUT PRE ACTION IN SUITE o.a.s.sql.SessionStateSuite = ``` Fixed them as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84907/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84907/testReport)** for PR 19893 at commit [`e2da334`](https://github.com/apache/spark/commit/e2da334560f764af71b6677f2e4c7c772063c229). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84907/testReport)** for PR 19893 at commit [`e2da334`](https://github.com/apache/spark/commit/e2da334560f764af71b6677f2e4c7c772063c229). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84879/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84879/testReport)** for PR 19893 at commit [`d38a0c6`](https://github.com/apache/spark/commit/d38a0c607d6a6d4a3e941115fae2db298adefa6e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84879/testReport)** for PR 19893 at commit [`d38a0c6`](https://github.com/apache/spark/commit/d38a0c607d6a6d4a3e941115fae2db298adefa6e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84869/testReport)** for PR 19893 at commit [`827b6dc`](https://github.com/apache/spark/commit/827b6dca818d3f678c74f0edda04a1f9b315f932). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84869/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @vanzin I've fixed the problematic tests and added a codepart in the ThreadAudit to highlight such situations. After the build we can see more. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @squito thanks for sharing your findings, it's helpful. Yeah, slowly digging into the deep and finding out what these threads are. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84869/testReport)** for PR 19893 at commit [`827b6dc`](https://github.com/apache/spark/commit/827b6dca818d3f678c74f0edda04a1f9b315f932). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 hi I'm just getting back, thanks for looking at this -- one quick comment on the discussion about why there needs to be a whitelist at all. I have a vague memory of looking at this a while back, and that even when you properly stop netty, all the cleanup is not done immediately, those threads linger around briefly for some reason. I dont' recall exactly which set of threads that applies to. anyway sounds like you have already figured this out independently, but just wanted to add my findings. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84858/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84858/testReport)** for PR 19893 at commit [`ef58576`](https://github.com/apache/spark/commit/ef58576d953ac98e29027477e9638e4508ee5ced). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84858/testReport)** for PR 19893 at commit [`ef58576`](https://github.com/apache/spark/commit/ef58576d953ac98e29027477e9638e4508ee5ced). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 You still have not looked at the hive tests. ``` $ grep -a 'POSSIBLE THREAD LEAK' unit-tests.log | wc -l 61 ``` A bunch of those look like false positives. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84763/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84763/testReport)** for PR 19893 at commit [`3bde33c`](https://github.com/apache/spark/commit/3bde33c481a35257497a1b9d4787bdc9ab7c7a3c). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait ThreadAudit extends Logging ` * `trait SharedSQLContext extends SQLTestUtils with SharedSparkSession ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84763/testReport)** for PR 19893 at commit [`3bde33c`](https://github.com/apache/spark/commit/3bde33c481a35257497a1b9d4787bdc9ab7c7a3c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @vanzin I've fixed the SQL test flow and additionally I've made the implementation less invasive by extracting the logic into a trait. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @jiangxb1987 feel free to take a look at it. More eyes, more possibilities. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19893 Oh, it's great that the shared context don't cause much false positives. My only concern now is that the number of warnings is scary, could you please give me one or two days to look into the (possible) leaking thread issues to make sure there is no other obvious false positives? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 @jiangxb1987 I don't know whether I understand your concern well but there is no intention to modify the shared `TestHiveContext ` among suites. It will remain as it is now. As an additional note, the plus functionality coming from this PR is only a printout which can be ignored any time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/19893 The `TestHiveContext` is shared among test suites, maybe it's not a good time to change this for now. Could we create a new class that inherit from `SparkFunSuite` that examines leaking threads, and only have test suites in core module extend from this new class? WDYT @gatorsmile @cloud-fan ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 I've analysed the hive related test flow and found SparkSession and SQLContext sharing between suites as you mentioned. Here is the execution flow: 1. The first hive test suite instantiates TestHive which creates SparkSession and SQLContext 2. SparkFunSuite.beforeAll creates a thread snapshot 3. Test code runs 4. TestHiveSingleton.afterAll resets SparkSession 5. SparkFunSuite.afterAll prints out the possible leaks Step one executed only by the first hive suite and never again. Here I do not see false positives in big scale. The only possible false positive threads what I foresee could come from lazy initialisation within SparkSession or SQLContext. On the leftover side we're not tracking SparkSession and SQLContext threads but because of the singleton nature my suggestion is to leave it like that. In this case you mentioned ``` $ grep 'POSSIBLE THREAD LEAK' unit-tests.log | wc -l 158 ``` I can imagine the following situations: 1. Test doesn't call hiveContext.reset() 2. Test creates thread but not frees up 3. Production code issue 4. ... Of course there could be other issues which I've not considered, please share your ideas. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Still to come. I'll put hive findings here the same way. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 > With this change I don't see any false positives and missed threads. That sounds good but it only covers sql. Did you also look at hive as I mentioned previously? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 As a next step analysed SQL test flow. Here are the steps: 1. SharedSparkSession.beforeAll called which initialise SparkSession and SQLContext 2. SparkFunSuite.beforeAll creates a thread snapshot 3. Test code runs 4. SparkFunSuite.afterAll prints out the possible leaks 5. SharedSparkSession.afterAll stops SparkSession Not sure if I understand right but this will not report false positives. The only problem what I see here as it's not gonna report SparkSession and SQLContext related leaks. As you mentioned before this code should find SparkContext related threading issues which applies here as well. This is not fulfilled at the moment and my proposal is to fix it this way: 1. SparkFunSuite.beforeAll creates a thread snapshot 2. SharedSparkSession.beforeAll called which initialise SparkSession and SQLContext 3. Test code runs 4. SharedSparkSession.afterAll stops SparkSession 5. SparkFunSuite.afterAll prints out the possible leaks With this change I don't see any false positives and missed threads. Please share your ideas related this topic. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 > Those try to keep the same session alive for multiple suites Good point to make this part clear. As a first step I've taken a look at the code and as I see SparkSession as well as SQLContext recreated between different suites. They're shared inside a suite. Have I missed something? ``` Class: org.apache.spark.sql.test.GenericFlatSpecSuite SparkSession: org.apache.spark.sql.test.TestSparkSession@1b5bc39d SQLContext: org.apache.spark.sql.SQLContext@655a5d9c Class: org.apache.spark.sql.test.GenericWordSpecSuite SparkSession: org.apache.spark.sql.test.TestSparkSession@53cddaf8 SQLContext: org.apache.spark.sql.SQLContext@55c50f52 Class: org.apache.spark.sql.test.GenericFunSpecSuite SparkSession: org.apache.spark.sql.test.TestSparkSession@22ff1372 SQLContext: org.apache.spark.sql.SQLContext@356341c9 Class: org.apache.spark.sql.test.DataFrameReaderWriterSuite SparkSession: org.apache.spark.sql.test.TestSparkSession@2c02a007 SQLContext: org.apache.spark.sql.SQLContext@63a72cc6 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 Yeah, I don't think this can be added before figuring out SQL tests. ``` $ grep 'POSSIBLE THREAD LEAK' unit-tests.log | wc -l 158 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84617/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84617/testReport)** for PR 19893 at commit [`644ee6a`](https://github.com/apache/spark/commit/644ee6aec14faa89789dd9ef52fb539e93a7076d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84617/testReport)** for PR 19893 at commit [`644ee6a`](https://github.com/apache/spark/commit/644ee6aec14faa89789dd9ef52fb539e93a7076d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 > All in all if you have an approach which solves context reuse feel free to share. I don't have one, but it feels sub-optimal to add code that will knowingly trigger false positives in a large number of existing test suites, so it should probably be handled somehow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19893 Yeah, this is fully true. This enhancement is definitely will not solve the issues once and for all. The problems were hidden till now and we would like to make a step ahead and make it at least visible somehow. With the help of this I've tracked down problems in additional 4-5 suites. All in all if you have an approach which solves context reuse feel free to share. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19893 Also, what happens when you run tests in the sql and hive modules? Those try to keep the same session alive for multiple suites, so I'd expect a bunch of threads to trigger this code when they really should not, in that case. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84601/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84601/testReport)** for PR 19893 at commit [`2b02d45`](https://github.com/apache/spark/commit/2b02d45303c018cae8e578804d07cf1803447c42). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84606/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #84606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84606/testReport)** for PR 19893 at commit [`62cb32b`](https://github.com/apache/spark/commit/62cb32b9058c3cf3aac4821cb3eecb83d6b577e7). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
