yuqi1129 opened a new issue, #12986:
URL: https://github.com/apache/gravitino/issues/12986
### Version
main branch
### Describe what's wrong
When a catalog is dropped or altered, its isolated ClassLoader should become
collectable. For `jdbc-*`, `kafka`, `model` and a `fileset` catalog on a local
filesystem it does. For `hive`, `lakehouse-iceberg`, `lakehouse-paimon` and a
`fileset` catalog backed by S3 it does not, and the classes stay in Metaspace
for the life of the process.
Because `alter` rebuilds the catalog's ClassLoader, a server that alters
catalogs accumulates one leaked loader per alter. Measured on a Hive catalog:
**five alters cost ~48 MB of Metaspace that is never returned**, so roughly 45
create/drop or alter operations exhaust a 512 MB `MaxMetaspaceSize`. The
failure mode is bad — with Metaspace full the JVM cannot load a class it has
not already loaded, so warm paths keep returning 200 while anything new returns
500, and the server looks like it has several unrelated feature bugs rather
than one memory problem.
Three distinct retention paths, each confirmed from a heap dump by walking
the reference graph back to a GC root:
**1. commons-logging registers a listener on the shared Log4j
`LoggerContext` (strong, permanent)**
```
AppClassLoader (JNI_GLOBAL root)
-> class org.slf4j.LoggerFactory --static PROVIDER--> SLF4JServiceProvider
-> Log4jLoggerFactory --registry-->
org.apache.logging.log4j.core.LoggerContext
-> listeners[2] =
org.apache.commons.logging.impl.Log4jApiLogFactory$LogAdapter
-> <class> --loader--> the catalog's ClassLoader
```
`LogFactory.release(loader)`, which `ClassLoaderResourceCleanerUtils`
already calls, drops the factory from its own cache but leaves this
registration in place.
**2. `ThreadLocal` values held behind a `SoftReference` (soft, released only
under heap pressure)**
```
Thread (GC root) -> threadLocals -> Entry.value = java.lang.ref.SoftReference
-> referent [SOFT] = ...jackson.core.util.BufferRecycler
-> <class> --loader--> the catalog's ClassLoader
```
`ClassLoaderResourceCleanerUtils.clearThreadLocalMap` tests
`value.getClass().getClassLoader()`. The value here is a `SoftReference`, a
bootstrap class, so the entry is never cleared and only the referent identifies
the catalog. This matters more than it sounds: soft references are cleared
under **heap** pressure, and Metaspace pressure never triggers that, so on a
server with a roomy heap and a small `MaxMetaspaceSize` these are effectively
permanent. The same method also returns early for any thread not named
`Gravitino-webserver-*`, while `catalog.close()` itself runs on a
`ForkJoinPool` worker.
**3. Hive's nested ClassLoader is never cleaned**
`HiveClientFactory` creates a second `HiveClientClassLoader` over
`hive-metastore{2,3}-libs` whose base loader is the catalog's own.
`HiveCatalogOperations.close()` passes only the catalog loader to
`ClassLoaderResourceCleanerUtils`, so the nested loader is closed but never
cleaned; anything pinning it pins the catalog loader through its
`baseClassLoader` field. Hadoop's `Shell` spawns sub-processes, and the JDK's
pooled `process reaper` threads inherit the spawning thread's context
ClassLoader — a GC root — which pins the nested loader until that pooled thread
idles out.
### Error message and/or stacktrace
```
java.lang.OutOfMemoryError: Metaspace
```
With Metaspace exhausted the server keeps running: already-loaded paths
answer normally, while any operation that needs a new class fails.
### How to reproduce
Start a server with `-XX:MaxMetaspaceSize=512m`, create a `hive` catalog
against a metastore, list its schemas, then drop it and force a full GC:
```bash
jcmd <pid> GC.run
jcmd <pid> VM.classloader_stats | grep -c CustomURLClassLoader
```
The catalog's `IsolatedClassLoader$CustomURLClassLoader` and its
`HiveClientClassLoader` are still listed, and `jcmd <pid> GC.heap_info` shows
Metaspace unchanged. Repeating with `alter` shows one retained loader per
alter. `lakehouse-paimon` (filesystem backend), `lakehouse-iceberg` (jdbc
backend) and `fileset` with `filesystem-providers=s3` reproduce it as well;
`jdbc-mysql`, `kafka`, `model` and a local-filesystem `fileset` do not.
Adding `-XX:SoftRefLRUPolicyMSPerMB=0` releases the Paimon, Iceberg and S3
loaders but not the Hive one, which separates path 2 from path 1.
### Additional context
Measured with one catalog at a time on a server started with `-Xms1024m
-Xmx1024m -XX:MaxMetaspaceSize=512m`:
| provider | drop, default GC flags | drop, soft refs cleared |
|---|---|---|
| `model`, `jdbc-mysql`, `kafka`, `fileset` (file://) | released | released |
| `fileset` (s3a, MinIO) | retained | released |
| `lakehouse-paimon` | retained | released |
| `lakehouse-iceberg` | retained | released |
| `hive` | retained | **retained** |
Related: a catalog whose initialization fails also leaves its loader behind.
Filed separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
