zabetak opened a new pull request, #3514: URL: https://github.com/apache/hive/pull/3514
### What changes were proposed in this pull request and why? * Add unit test to trigger the memory leak (not meant to be committed in the repo cause it is a bit hacky and slow) * (Main fix) Ensure FileSystem resources are closed correctly by wrapping the code in try-finally block. In order for the issue to appear the following requirements must be met: * Secure HMS/HDFS communication * Many compaction failures when removing files * Many users doas running compactions More info under HIVE-26404. ### Does this PR introduce _any_ user-facing change? Fixes the memory leak. ### How was this patch tested? Run the new test with and without the fix. Without the fix the test: * it will either hang and there will be `OutOfMemory` in `target/surefire-reports/org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithSecureDFS-output.txt` * it will finish after a long time and there will be `OutOfMemory` in `target/surefire-reports/org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithSecureDFS-output.txt` To confirm the memory leak it is the same with the one obseved in HIVE-26404 I took a heapdump when the test was running and validated that there are many `FileSystem` objects in the cache each holding `Configuration` objects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
