yuqi1129 opened a new pull request, #12987:
URL: https://github.com/apache/gravitino/pull/12987
### What changes were proposed in this pull request?
- `ClassLoaderResourceCleanerUtils.clearThreadLocalMap` now looks
**through** a `java.lang.ref.Reference` value to its referent when deciding
whether a thread-local entry belongs to the dying loader, also checks the
entry's key (the `ThreadLocal` itself), and no longer skips threads whose name
is not `Gravitino-webserver-*`.
- New cleanup step `removeLoggerContextListeners`, which removes listeners
the loader registered on the shared Log4j `LoggerContext`.
- `HiveClientFactory.close()` runs the resource cleaner against the nested
`HiveClientClassLoader` before closing it.
### Why are the changes needed?
Dropping or altering a `hive`, `lakehouse-iceberg`, `lakehouse-paimon`, or
S3-backed `fileset` catalog leaked its ClassLoader, so Metaspace grew until the
JVM could no longer load new classes and the server degraded into per-feature
500s while already-warm paths kept returning 200.
The three retention paths, each traced from a heap dump back to a GC root,
are described in detail in the issue.
Fix: #12986
### Does this PR introduce _any_ user-facing change?
No new configuration or API. Dropped catalogs release their Metaspace, so a
long-running server no longer grows without bound.
### How was this patch tested?
Five new unit tests in `TestClassLoaderResourceCleanerUtils` cover looking
through a reference, ignoring cleared references, clearing a soft-referenced
thread-local value, and leaving unrelated entries alone.
`:catalogs:catalog-common`, `:catalogs:hive-metastore-common`,
`:catalogs:catalog-hive` and `:catalogs:catalog-fileset` pass (53 tests in the
two directly affected modules).
End-to-end on a packaged server with `-Xms1024m -Xmx1024m
-XX:MaxMetaspaceSize=512m`, one catalog at a time, dropping and forcing a full
GC, against a real Hive metastore, MySQL, Kafka and MinIO in containers. Loader
counts come from `jcmd VM.classloader_stats`, Metaspace from `jcmd
GC.heap_info`:
| scenario | before | after |
|---|---|---|
| hive create + drop | 3 loaders retained | 0 retained, Metaspace 60.1 →
51.7 MB |
| hive, 5 × alter | 7 loaders, 70.6 → 119.3 MB, permanent | back to
baseline, +0.8 MB |
| paimon create + table + drop | retained | released, 61.2 → 54.6 MB |
| iceberg create + table + drop | retained | released, 76.4 → 67.4 MB |
| fileset on s3a create + fileset + drop | retained | released, 68.2 → 53.4
MB |
`jdbc-mysql`, `kafka`, `model` and a local-filesystem `fileset` released
their loaders before this change and still do.
```bash
./gradlew :catalogs:catalog-common:test :catalogs:hive-metastore-common:test
\
:catalogs:catalog-hive:test :catalogs:catalog-fileset:test -PskipITs
-PskipWeb=true
```
https://claude.ai/code/session_013xVSteM2ZUjXRHFbHtayVK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
