[ https://issues.apache.org/jira/browse/SPARK-16139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-16139. ------------------------------------ Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 19893 [https://github.com/apache/spark/pull/19893] > Audit tests for leaked threads > ------------------------------ > > Key: SPARK-16139 > URL: https://issues.apache.org/jira/browse/SPARK-16139 > Project: Spark > Issue Type: Test > Components: Tests > Affects Versions: 2.0.0 > Reporter: Imran Rashid > Assignee: Gabor Somogyi > Priority: Major > Fix For: 2.4.0 > > > Lots of our tests don't properly shutdown everything they create, and end up > leaking lots of threads. For example, {{TaskSetManagerSuite}} doesn't stop > the extra {{TaskScheduler}} and {{DAGScheduler}} it creates. There are a > couple more instances I've run into recently, eg. in > [{{DAGSchedulerSuite}}|https://github.com/apache/spark/commit/cf1995a97645f0b44c997f4fdbba631fd6b91a16#diff-f3b410b16818d8f34bb1eb4120a60d51R235 > ] > I'm fixing these piecemeal when I see them (eg., TaskSetManagerSuite should > be fixed by my pr for SPARK-16136), but it would be great to have a > comprehensive audit and fix this across all tests. > This should be semi-automatable. In {{SparkFunSuite}}, you could grab all > threads before the tests starts, and after it completes. Then you could > clearly log all threads started after the test started but still going. > Unfortunately this isn't perfect, it seems that netty threads aren't killed > immediately on shutdown, . Its OK if some of them linger beyond the test, so > you may need to do some whitelisting based on thread-name & a little more > manual inspection. But you could at least clearly log the relevant info, so > that after a jenkins run you could easily pull the info from the logs. > Bonus points if you can figure out some way to make this output visible > outside of the logs, ideally even in the test report that makes it to github, > but that isn't necessary, and unless its very easy probably best for a > separate task. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org