yuqi1129 opened a new issue, #13016:
URL: https://github.com/apache/gravitino/issues/13016
### What would you like to be improved?
The Glue catalog does not release its isolated ClassLoader when it is
dropped, so its classes stay in Metaspace for the life of the process. Every
other provider I could run locally releases its loader after #12987; Glue is
the one that does not, and its cause is different in kind, which is why it is
filed on its own.
Verified against LocalStack on a packaged server started with `-Xms1024m
-Xmx1024m -XX:MaxMetaspaceSize=512m`: create a `glue` catalog, list and create
a schema, drop the catalog, then force a full GC. `jcmd <pid>
VM.classloader_stats` still lists the catalog's
`IsolatedClassLoader$CustomURLClassLoader` with ~2265 classes, and `jcmd <pid>
GC.heap_info` shows Metaspace unchanged.
Tracing the heap dump back to a GC root gives:
```
JAVA_FRAME (a live thread's stack)
->
software.amazon.awssdk.http.apache.internal.conn.IdleConnectionReaper$ReaperTask
-> <class> --loader--> the catalog's ClassLoader
```
The thread is `idle-connection-reaper`, running
`IdleConnectionReaper$ReaperTask.run`. The AWS SDK keeps that reaper as a
singleton per ClassLoader and stops it only when the last connection manager is
deregistered, that is, when every Apache-HTTP-backed client built by that
loader has been closed. It also swallows interrupts, so the ClassLoader cleanup
cannot stop it from the outside: I confirmed the cleanup does interrupt the
thread and the thread keeps running.
`GlueCatalogOperations.close()` does close its `glueClient`, so the reaper
staying alive says some other AWS client the catalog builds is never closed.
The warehouse is an `s3a://` location, so an S3 client is the obvious
candidate, but I did not chase which one.
### How should we improve?
Find the AWS client that outlives `GlueCatalogOperations.close()` and close
it there, so the SDK deregisters its connection manager and the reaper stops on
its own.
Please do not paper over this in `ClassLoaderResourceCleanerUtils` by
reflecting into `IdleConnectionReaper` and forcing the thread down. That would
hide a client that is leaking sockets and connection pools for as long as the
server runs, which is a problem in its own right, and it would depend on SDK
internals that are not API.
A regression test in the same shape as the manual check would be worth
having: drop the catalog, force a full GC, assert that no
`CustomURLClassLoader` for it remains.
Found while working on #12986 / #12987, which fix the same symptom for
`hive`, `lakehouse-iceberg`, `lakehouse-paimon`, the JDBC catalogs and
`fileset` on s3a, gs and abfss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
