Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/19893#discussion_r160158067
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala ---
@@ -17,4 +17,22 @@
package org.apache.spark.sql.test
-trait SharedSQLContext extends SQLTestUtils with SharedSparkSession
+trait SharedSQLContext extends SQLTestUtils with SharedSparkSession {
+
+ /**
+ * Auto thread audit is turned off here intentionally and done manually.
--- End diff --
Same thing but it meant to solve different problem (changes the execution
order). Please see the execution order with and without this change described
in my previous post:
```
As a next step analysed SQL test flow. Here are the steps:
1. SharedSparkSession.beforeAll called which initialise SparkSession and
SQLContext
2. SparkFunSuite.beforeAll creates a thread snapshot
3. Test code runs
4. SparkFunSuite.afterAll prints out the possible leaks
5. SharedSparkSession.afterAll stops SparkSession
Not sure if I understand right but this will not report false positives.
The only problem what I see here as it's not gonna report SparkSession and
SQLContext related leaks.
As you mentioned before this code should find SparkContext related
threading issues which applies here as well. This is not fulfilled at the
moment and my proposal is to fix it this way:
1. SparkFunSuite.beforeAll creates a thread snapshot
2. SharedSparkSession.beforeAll called which initialise SparkSession and
SQLContext
3. Test code runs
4. SharedSparkSession.afterAll stops SparkSession
5. SparkFunSuite.afterAll prints out the possible leaks
With this change I don't see any false positives and missed threads.
Please share your ideas related this topic.
```
Your concern is fully standing but this change is not intended to cover the
mentioned issue. The problem you mentioned is addressed in ThreadAudit, namely
it prints out the following message such cases:
```
THREAD AUDIT POST ACTION CALLED WITHOUT PRE ACTION IN SUITE...
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
