sunchao commented on code in PR #7082:
URL: https://github.com/apache/datafusion-comet/pull/7082#discussion_r4356781653
##########
native/core/src/execution/jni_api.rs:
##########
@@ -461,6 +464,7 @@ pub fn get_runtime() -> Handle {
/// would deadlock/panic.
pub fn release_runtime() {
crate::execution::operators::clear_file_io_cache();
+ crate::parquet::shared_metadata_cache::clear_shared_metadata_caches();
Review Comment:
[P2] Run cache cleanup after restarted SparkContexts too. Start and stop a
local Comet SparkContext, then start another in the same JVM, scan Parquet, and
stop it. Both plugins call `NativeBase.releaseNative()`, but its process-wide
`released` flag remains true after the first shutdown, so the second shutdown
never reaches this new cleanup. The new registry consequently retains up to 100
MiB of metadata after all plans and the second context have gone, until later
cache eviction or process exit. Before this PR, private metadata caches were
freed with their plans without depending on this hook. Please make cleanup run
for each plugin lifecycle, with a regression that verifies release after two
context lifecycles.
Evidence: A bounded JNI component probe used the unchanged exact-head
`NativeBase.java`, bootstrap-only dependency stubs, and the actual
`Java_org_apache_comet_NativeBase_release` entry point. For each of two
application keys it inserted a 16 MiB metadata payload into the real shared
registry, dropped the task handle, and called `releaseNative()`. Results: first
call retained 0 bytes; second call retained 16,777,216 bytes; directly calling
the native `release()` then retained 0 bytes. Source trace:
`Plugins.scala:151,423` calls `releaseNative`; `NativeBase.java:327` gates it
with `released.compareAndSet(false, true)`, with no reset. Probe patch, Java
sources, and output are preserved under `/tmp/comet7082-d80-release-probe/`. No
full SparkContext restart application was executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
