[
https://issues.apache.org/jira/browse/OPENJPA-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125438#comment-18125438
]
ASF subversion and git services commented on OPENJPA-3012:
----------------------------------------------------------
Commit 921891068d9ec1de422f10515bef11dfb9b29407 in openjpa's branch
refs/heads/master from Paulo Cristovão de Araújo Silva Filho
[ https://gitbox.apache.org/repos/asf?p=openjpa.git;h=921891068 ]
Merge pull request #210 from apache/OPENJPA-3012
[OPENJPA-3012] no new classloaders are being registered in
ClassResolvingObjectInputStream.resolveClass
> memory leak in Serialization
> ----------------------------
>
> Key: OPENJPA-3012
> URL: https://issues.apache.org/jira/browse/OPENJPA-3012
> Project: OpenJPA
> Issue Type: Bug
> Components: datacache, kernel
> Affects Versions: 3.2.2
> Reporter: Maxim Solodovnik
> Assignee: Maxim Solodovnik
> Priority: Major
> Fix For: 4.2.0
>
>
> h2. Summary
> org.apache.openjpa.util.Serialization.ClassResolvingObjectInputStream.resolveClass()
> creates a new MultiClassLoader instance for every class descriptor it
> resolves, and passes it to Class.forName(name, true, loader).
> When a loader is used as the initiating loader in Class.forName, the JVM
> registers it and allocates native ClassLoaderData structures for it. These
> show up in Native Memory Tracking (NMT) under the "Class", "Module" and
> "Synchronization" categories. The throw-away loaders become garbage
> immediately, but in our environment their native memory was only reclaimed by
> a Full GC. G1 concurrent cycles did not reclaim it.
> TCPRemoteCommitProvider uses this stream to deserialize every incoming
> RemoteCommitEvent. In a clustered deployment, every node therefore creates
> several new class loaders for every commit made on any other node. Under
> sustained write load this caused steady native memory growth on all nodes,
> until the containers were OOMKilled. The Java heap was not the problem: it
> still had plenty of free space when the containers were killed.
> h2. How we use OpenJPA
> * OpenJPA is the JPA provider, with the L2 DataCache enabled
> (openjpa.DataCache=true).
> * Cache coherence between replicas uses the Kubernetes remote commit
> provider shipped in openjpa-kubernetes (introduced by OPENJPA-2801):
> openjpa.RemoteCommitProvider=org.apache.openjpa.event.kubernetes.KubernetesTCPRemoteCommitProvider(CacheDurationMillis=30000,Namespace=<ns>,Label=<label>)
> * Each replica sends a RemoteCommitEvent to its peers on every commit. Each
> replica receives the events of all other replicas through
> TCPRemoteCommitProvider$TCPPortListener$ReceiveSocketHandler.
> * The workload is write-heavy (bulk provisioning): hundreds of thousands of
> entity inserts and updates over 15+ hours, spread across all replicas.
> h2. Symptoms
> # Memory grew on every replica at about the same rate (~450 MB/hour). This
> was true even for replicas with almost no local work. Every replica receives
> the remote commit events of all the others, so the growth followed the
> cluster-wide commit rate, not each replica's own load.
> # Containers were repeatedly OOMKilled (exit code 137) while the Java heap
> was far below its maximum.
> * Example: container RSS reached 24.3 GB (the limit) while the Java heap was
> 11.4 GB, out of a 17.2 GB maximum.
> * Increasing the container memory or tuning GC settings did not help.
> # Some replicas were restarted by their liveness probe after long JVM pauses.
> * This happened when a large backlog of ClassLoaderData was finally
> processed.
> * It also happened when we ran diagnostic commands that walk all class
> loaders (jcmd VM.classloader_stats).
> h2. Evidence
> h3. 1. NMT summary of a replica after ~11 hours
> (-XX:NativeMemoryTracking=summary)
> {code:java}
> Total: reserved=26540MB, committed=12155MB
> malloc: 7024MB #31316799
> - Java Heap (reserved=17216MB, committed=4624MB)
> - Class (reserved=4439MB, committed=3434MB)
> (classes #28771)
> (malloc=3415MB #12733544) (peak=4961MB #18457246)
> ( Metadata: )
> ( reserved=192MB, committed=142MB)
> ( used=140MB)
> - Module (reserved=2545MB, committed=2545MB)
> (malloc=2545MB #7582516) (peak=3697MB #11015426)
> - Synchronization (reserved=251MB, committed=251MB)
> (malloc=251MB #2528005)
> - Symbol (reserved=222MB, committed=222MB)
> - Metaspace (reserved=193MB, committed=143MB)
> {code}
> * No class loading problem. The number of loaded classes (~28.8K) and
> Metaspace (~143 MB) were stable.
> * Growth was in malloc only. It was millions of small native allocations in
> the Class, Module and Synchronization categories. Together they used more
> memory than the committed Java heap.
> * The allocation ratio points to ClassLoaderData. The ratio of allocations
> (Class : Module : Synchronization ≈ 5 : 3 : 1) matches what the JVM allocates
> for each registered class loader: dictionary, unnamed module and package
> tables, and the metaspace lock.
> h3. 2. The throw-away loaders are MultiClassLoader instances
> About 5 minutes after a replica restart, under load:
> $ jcmd <pid> VM.classloader_stats | grep -c MultiClassLoader 14091
> That is about 47 new org.apache.openjpa.lib.util.MultiClassLoader loaders per
> second.
> h3. 3. Only a Full GC reclaims the memory
> ||Step||NMT Class (committed)||NMT Module (committed)||
> |Fresh replica, before|36 MB|15 MB|
> |jcmd <pid> GC.run (with -XX:+ExplicitGCInvokesConcurrent, so a G1 concurrent
> cycle), +2 min|50 MB|26 MB|
> |Some minutes later|72 MB|43 MB|
> |jcmd <pid> GC.class_histogram (forces a Full GC)|17 MB|1 MB|
> * After the Full GC, the histogram showed 0 live MultiClassLoader instances.
> The loaders are unreachable garbage, not a reference leak.
> * However, their native memory was not released by the G1 concurrent cycle
> (Remark took place, as shown in the GC log). It was only released by the Full
> GC.
> * In a long-running service, Full GCs are rare, so the memory accumulates.
> h3. 4. Standalone reproduction (no application code)
> {code:java}
> import java.io.*;
> import java.util.*;
> import org.apache.openjpa.util.Serialization;
> public class Repro {
> public static void main(String[] args) throws Exception {
> ArrayList<Object> value = new ArrayList<>();
> value.add(new HashMap<>(Map.of("k", 1L)));
> value.add(new HashSet<>(Set.of("x")));
> value.add(new Date());
> ByteArrayOutputStream bytes = new ByteArrayOutputStream();
> new ObjectOutputStream(bytes).writeObject(value);
> byte[] data = bytes.toByteArray();
> for (int i = 0; i < 50_000; i++) {
> new Serialization.ClassResolvingObjectInputStream(new
> ByteArrayInputStream(data)).readObject();
> }
> Thread.sleep(5_000); // inspect with: jcmd <pid> VM.native_memory
> summary / VM.classloader_stats
> }
> }
> {code}
> Run with {{java -XX:NativeMemoryTracking=summary -XX:+UseG1GC -Xmx512m -cp
> openjpa-3.2.2.jar:.}} Repro. Results with JDK 17:
> || ||NMT Class malloc||NMT Module malloc||MultiClassLoader loaders
> registered||
> |OpenJPA 3.2.2|168 MB (606,620 allocations)|125 MB (363,310
> allocations)|92,308|
> |With the patch below|0.1 MB|0.03 MB|0|
> h2. Root cause
> {{{}openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java{}}}:
> * 3.2.2:
> [https://github.com/apache/openjpa/blob/3.2.2/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java#L131-L141]
> * master:
> [https://github.com/apache/openjpa/blob/master/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java#L122-L131]
> {code:java}
> @Override
> protected Class resolveClass(ObjectStreamClass desc)
> throws IOException, ClassNotFoundException {
> String name = BlacklistClassResolver.DEFAULT.check(desc.getName());
> MultiClassLoader loader = AccessController
> .doPrivileged(J2DoPrivHelper.newMultiClassLoaderAction()); // new
> loader per class descriptor
> addContextClassLoaders(loader);
> loader.addClassLoader(getClass().getClassLoader());
> loader.addClassLoader(MultiClassLoader.SYSTEM_LOADER);
> return Class.forName(name, true, loader); // JVM
> registers the new loader (native ClassLoaderData)
> }
> {code}
> {{resolveClass()}} is called once for each class descriptor in the stream. A
> single RemoteCommitEvent contains several class descriptors: the event class,
> collections, and OpenJPA id types. So each received event registers several
> new loaders with the JVM.
> The hot caller is
> {{{}TCPRemoteCommitProvider$TCPPortListener$ReceiveSocketHandler{}}}:
> * 3.2.2:
> [https://github.com/apache/openjpa/blob/3.2.2/openjpa-kernel/src/main/java/org/apache/openjpa/event/TCPRemoteCommitProvider.java#L712-L726]
> * master:
> [https://github.com/apache/openjpa/blob/master/openjpa-kernel/src/main/java/org/apache/openjpa/event/TCPRemoteCommitProvider.java#L704-L718]
> {code:java}
> ObjectInputStream ois = new Serialization.ClassResolvingObjectInputStream(in);
> ...
> byte[] senderAddress = (byte[]) ois.readObject();
> RemoteCommitEvent rce = (RemoteCommitEvent) ois.readObject();
> Serialization.deserialize(...)
> {code}
> is also used by {{DBDictionary.getBlobObject()}} for serialized BLOB fields,
> so applications that use such fields are affected in the same way.
> On {{{}master{}}}, the {{doPrivileged}} wrapper was removed (OPENJPA-2940),
> but {{resolveClass()}} still does {{new MultiClassLoader()}} per call. *The
> issue is still present.*
> h2. Fix we applied (tested in production-like load)
> We changed only ClassResolvingObjectInputStream. The candidate loaders are
> collected once per stream, in the same order as before. Classes are then
> resolved directly against those existing loaders, so no new class loader is
> ever passed to the JVM.
> * The protected addContextClassLoaders(MultiClassLoader) hook is kept, so
> PersistentObjectInputStream (which adds the StoreContext loader) works
> unchanged.
> * The MultiClassLoader instance is used only as an ordered, de-duplicating
> container. It is never handed to Class.forName, so the JVM never registers it.
> {code}
> --- a/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java
> (3.2.2)
> +++ b/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java
> @@ public static class ClassResolvingObjectInputStream extends
> ObjectInputStream {
>
> + /**
> + * Candidate loaders, in the same order the original MultiClassLoader
> + * used them; resolved lazily once per stream.
> + */
> + private ClassLoader[] _loaders;
> +
> public ClassResolvingObjectInputStream(InputStream delegate)
> throws IOException {
> super(delegate);
> }
> @Override
> protected Class resolveClass(ObjectStreamClass desc)
> throws IOException, ClassNotFoundException {
> String name =
> BlacklistClassResolver.DEFAULT.check(desc.getName());
> - MultiClassLoader loader = AccessController
> - .doPrivileged(J2DoPrivHelper.newMultiClassLoaderAction());
> - addContextClassLoaders(loader);
> - loader.addClassLoader(getClass().getClassLoader());
> - loader.addClassLoader(MultiClassLoader.SYSTEM_LOADER);
> - return Class.forName(name, true, loader);
> +
> + ClassNotFoundException notFound = null;
> + for (ClassLoader loader : getCandidateLoaders()) {
> + try {
> + return Class.forName(name, true, loader);
> + } catch (ClassNotFoundException e) {
> + if (notFound == null)
> + notFound = e;
> + }
> + }
> +
> + // primitive types and anything the candidates could not see
> + try {
> + return super.resolveClass(desc);
> + } catch (ClassNotFoundException e) {
> + throw notFound != null ? notFound : e;
> + }
> + }
> +
> + private ClassLoader[] getCandidateLoaders() {
> + if (_loaders == null) {
> + // MultiClassLoader is used here only as an ordered,
> de-duplicating
> + // container so that subclasses overriding
> addContextClassLoaders keep
> + // working; it is never handed to the JVM, hence no
> ClassLoaderData
> + // is created for it.
> + MultiClassLoader holder = new MultiClassLoader();
> + addContextClassLoaders(holder);
> + holder.addClassLoader(getClass().getClassLoader());
> + holder.addClassLoader(MultiClassLoader.SYSTEM_LOADER);
> + _loaders = java.util.Arrays.stream(holder.getClassLoaders())
> + .filter(java.util.Objects::nonNull)
> + .distinct()
> + .toArray(ClassLoader[]::new);
> + }
> + return _loaders;
> }
> {code}
> h3. Notes on behavior
> * Lookup order is unchanged. The order is: thread context class loader, then
> (for PersistentObjectInputStream) the StoreContext loader, then the stream's
> own class loader, then the system class loader. Bootstrap classes are found
> through normal parent delegation of each candidate.
> * Initialization is unchanged. initialize = true is kept, as in the original.
> * Added fallback. If no candidate can load the class, we fall back to
> ObjectInputStream.resolveClass(desc), which also handles primitive types. If
> that also fails, the first ClassNotFoundException is rethrown.
> * Loaders are fixed per stream. The candidate list is computed once per
> stream instance, so the context class loader is captured on the first
> resolveClass call. For TCPRemoteCommitProvider the stream lives on a single
> receiver thread, so this is equivalent to the original behavior.
> h3. Result
> * The reproduction above no longer creates any class loader (see the table
> in Evidence 4).
> * In a repeated 15+ hour load test on the same 6-replica cluster, the native
> memory growth and the OOMKilled restarts no longer occur.
> h3. Possible alternatives for the upstream fix
> * Cache a single MultiClassLoader per stream (or per set of loaders) and
> reuse it, instead of creating one per resolveClass() call. This still
> registers one loader per stream, so it is less effective than resolving
> directly against the candidate loaders.
> * Review other places where a MultiClassLoader is created per call and then
> passed to Class.forName. OPENJPA-2057 notes that class-loading decisions are
> spread across the code base.
> ----
> h2. Related existing issues (no exact duplicate found)
> Searched on 2026-10-07 for MultiClassLoader, ClassResolvingObjectInputStream,
> TCPRemoteCommitProvider, classloader leak, native memory, Metaspace and
> ClassLoaderData.
> ||Issue||Status||Relation||
> |OPENJPA-2454 Memory leak when using multiple classloaders|Open (2.2.2)|Also
> a class-loader-related memory leak, but caused by application-created loaders
> retaining classes. Different mechanism.|
> |OPENJPA-2057 Rethinking ClassLoading architecture|Open|Notes that
> class-loading strategy is spread across many places, including ad-hoc
> MultiClassLoader usage. This report is one concrete runtime consequence.|
> |OPENJPA-1755 MultiClassLoader relies on its underlying ClassLoaders for
> hashCode|Open|Unrelated defect in MultiClassLoader. Its discussion assumes
> MultiClassLoaders are not created in large numbers, which is not true on this
> code path.|
> |OPENJPA-2801 Kubernetes TCPRemoteCommitProvider|Resolved|Introduced the
> provider we use. The leak is in the shared TCP receive path, so it affects
> all TCPRemoteCommitProvider subclasses.|
> |OPENJPA-2538, OPENJPA-2636, OPENJPA-285|Closed|Earlier class loader leaks
> (PCRegistry, custom plugins). Different code paths.|
> ----
> h2.
> Best regards,
> Greta Jang
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
