DanielLeens opened a new pull request, #12626: URL: https://github.com/apache/seatunnel/pull/12626
### Purpose of this pull request Fixes the worker `OutOfMemoryError: Metaspace` reported in #12456 for repeated MongoDB -> Hive jobs with `classloader-cache-mode=false`. **Root cause (from the heap dumps in the issue, 2.3.13).** `DefaultClassLoaderService` correctly removes a job class loader when its reference count reaches zero (the `classLoaderCache` is empty after the job). The released `SeaTunnelChildFirstClassLoader`, and the Metaspace of every class it defined, is nevertheless never collected because two library-level roots still point to it: 1. **Hadoop `WritableComparator.comparators`** (static registry, GC root = `sun.misc.Launcher$AppClassLoader`). `WritableComparator.get(Class, Configuration)` stores the caller's `Configuration` into the shared comparator singleton, and a `Configuration` captures the thread context class loader (the job loader) at construction. `org.apache.hadoop` is always parent-first in `SeaTunnelChildFirstClassLoader`, so the registry lives in the application class loader and survives every job: `Text$Comparator.conf -> Configuration.classLoader -> SeaTunnelChildFirstClassLoader`. 2. **MongoDB driver `PowerOfTwoBufferPool.DEFAULT`**. Its static initializer starts a `BufferPoolPruner-*` thread that never exits. The live thread pins the loader through its `ThreadFactory` class (`DaemonThreadFactory`) and through its `inheritedAccessControlContext`. `recycleClassLoaderFromThread` only resets the thread context class loader, which does not cover these two edges. Both roots are still present on current `dev` (`origin/dev` at `b0dd4945b1`): `DefaultClassLoaderService.releaseClassLoader` has no handling for either, `connector-mongodb` still pins `mongodb-driver-core` 4.7.1 (which has `PowerOfTwoBufferPool.DEFAULT` + `disablePruning()`), and the `seatunnel-hadoop3-3.1.4-uber` / `3.3.6-uber` jars still have the `comparators` / `conf` layout. No open PR addresses these roots; #10678 adds an opt-in `URLClassLoader.close()` deep clean, which does not remove either static/thread reference. **What this PR changes.** A small, isolated `ReleasedClassLoaderReferenceCleaner` runs once, at the point where `releaseClassLoader` physically drops the last reference (never in cache mode): - Mongo: if `PowerOfTwoBufferPool` is **defined by the released loader**, call the driver's own `disablePruning()` so the pruner executor shuts down and the thread exits. - Hadoop: for every `WritableComparator` registration, reset the shared comparator's `Configuration` **only if that `Configuration` belongs to the released loader** (back to the pristine `null` state a never-used comparator has), and remove registrations whose key/value class is defined by the released loader. Safety properties (the points raised in the issue discussion): - Nothing process-wide is cleared. A Mongo pool defined by an ancestor loader (shared by other jobs) is never stopped; a comparator whose `Configuration` belongs to another, still running loader is untouched; the registrations themselves are kept. - Library classes are resolved by name through the released loader with `Class.forName(name, false, loader)`, so the engine needs no Hadoop/Mongo dependency and the step is a no-op when a library is absent. - Best effort: any failure is logged and can never prevent the class loader from being released. **Scope / not covered.** This PR only covers the two roots with direct MAT evidence in #12456. Other Hadoop statics keyed by job classes (for example `CodecPool`, `ReflectionUtils` constructor cache, RPC client `Configuration`, token renewer `ServiceLoader`, reported by a contributor's out-of-tree agent in the same thread) are intentionally not touched until a heap dump shows them on a current build. `classloader-cache-mode=true` remains a valid mitigation for identical jar sets. **Design note for reviewers.** `AGENTS.md` asks to avoid connector-specific logic in the engine. This change keeps it to one package-private class plus a one-line call. The cleaner is small and has no dependency on the libraries, but if maintainers prefer, the alternative is a connector-owned release hook (SPI discovered with `ServiceLoader` on the released loader), which needs a new public API and is a larger change. Happy to rework in that direction. ### Does this PR introduce _any_ user-facing change? No configuration, API or default changed. Behavior change: with `classloader-cache-mode=false`, job class loaders that used the MongoDB or Hadoop (e.g. Hive/HDFS file sink) connectors become collectable after the job ends instead of accumulating in Metaspace. ### How was this patch tested? Added `ReleasedClassLoaderReferenceCleanerTest` (engine-core). Fixtures mirror the exact shapes found in the heap dump (static `DEFAULT` + package-private `disablePruning()`; static `comparators` + `getConf/setConf` + class-loader-capturing configuration). | Test | Production regression it turns red for | |---|---| | `stopsPrunerOfPoolDefinedByReleasedLoader` | the Mongo pruner is not stopped, or the wrong pool object is used | | `keepsPrunerOfPoolDefinedByAncestorLoader` | the ownership guard is dropped and a shared, process-wide pool is stopped under other jobs | | `detachesOnlyConfigurationOfReleasedLoader` | the released job's `Configuration` is not reset, or a running job's `Configuration`/registration is touched | | `removesRegistrationsKeyedByClassOfReleasedLoader` | registrations keyed by a class of the released loader keep pinning it, or unrelated registrations are removed | | `serviceCleansOnlyWhenLastReferenceIsReleased` | the hook is not wired into `releaseClassLoader`, or fires while another task still references the loader | | `serviceKeepsReferencesOfSharedLoaderInCacheMode` | cache mode (shared, never released loader) gets cleaned | Local verification was limited to `./mvnw spotless:apply -pl seatunnel-engine/seatunnel-engine-core`. Compilation and the tests above are verified by GitHub CI only. Not verified: an end-to-end MongoDB -> Hive reproduction with heap-dump comparison before/after on this branch; the reporter's environment (2.3.13, JDK 8u202) is the only place the leak has been observed so far. ### Check list * [x] No new Jar binary package. * [x] No documentation change needed (no user-facing option). * [x] No incompatible change. Related: #12456, #10669 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
