shivadarshan-devadiga commented on issue #12858:
URL: https://github.com/apache/gluten/issues/12858#issuecomment-6796444850

   @LuciferYang I looked into this, and I don't think the cached relations 
outlive the test.
   
   `GlutenDataFrameFunctionsSuite` extends Spark's `DataFrameFunctionsSuite`, 
which uses `SharedSparkSession`. `SharedSparkSessionBase.afterEach()` calls 
`spark.sharedState.cacheManager.clearCache()` after every test (Spark 3.4, 3.5, 
4.0 and master). None of the Gluten traits mixed into this suite override 
`afterEach`, in any of the five modules. So the cache is cleared when `flatten 
function` finishes, and later cases can't see it.
   
   I checked it at runtime on main (Velox, Spark 3.5) by temporarily overriding 
`afterEach` in `GlutenDataFrameFunctionsSuite` to print the cache state around 
`super.afterEach()`, then running only `Gluten - flatten function`:
   
   ```
   - Gluten - flatten function
   CACHECHECK cacheEmptyBeforeAfterEach=false cacheEmptyAfterAfterEach=true
   ```
   
   The relations are cached while the test runs, which is what it tests 
(codegen over a cached relation), and gone after it. Spark's own `flatten 
function` and `transform keys` tests in `DataFrameFunctionsSuite` cache the 
same way without `unpersist()`, relying on the same cleanup. Adding 
`unpersist()` here would not change behaviour, so I think this issue can be 
closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to