Maxim Solodovnik created OPENJPA-3012:
-----------------------------------------
Summary: memory leak in Serialization
Key: OPENJPA-3012
URL: https://issues.apache.org/jira/browse/OPENJPA-3012
Project: OpenJPA
Issue Type: Bug
Components: datacache, kernel
Affects Versions: 3.2.2
Reporter: Maxim Solodovnik
Assignee: Maxim Solodovnik
Fix For: 4.2.0
h2. Summary
org.apache.openjpa.util.Serialization.ClassResolvingObjectInputStream.resolveClass()
creates a new MultiClassLoader instance for every class descriptor it
resolves, and passes it to Class.forName(name, true, loader).
When a loader is used as the initiating loader in Class.forName, the JVM
registers it and allocates native ClassLoaderData structures for it. These show
up in Native Memory Tracking (NMT) under the "Class", "Module" and
"Synchronization" categories. The throw-away loaders become garbage
immediately, but in our environment their native memory was only reclaimed by a
Full GC. G1 concurrent cycles did not reclaim it.
TCPRemoteCommitProvider uses this stream to deserialize every incoming
RemoteCommitEvent. In a clustered deployment, every node therefore creates
several new class loaders for every commit made on any other node. Under
sustained write load this caused steady native memory growth on all nodes,
until the containers were OOMKilled. The Java heap was not the problem: it
still had plenty of free space when the containers were killed.
h2. How we use OpenJPA
* OpenJPA is the JPA provider, with the L2 DataCache enabled
(openjpa.DataCache=true).
* Cache coherence between replicas uses the Kubernetes remote commit provider
shipped in openjpa-kubernetes (introduced by OPENJPA-2801):
openjpa.RemoteCommitProvider=org.apache.openjpa.event.kubernetes.KubernetesTCPRemoteCommitProvider(CacheDurationMillis=30000,Namespace=<ns>,Label=<label>)
* Each replica sends a RemoteCommitEvent to its peers on every commit. Each
replica receives the events of all other replicas through
TCPRemoteCommitProvider$TCPPortListener$ReceiveSocketHandler.
* The workload is write-heavy (bulk provisioning): hundreds of thousands of
entity inserts and updates over 15+ hours, spread across all replicas.
h2. Symptoms
# Memory grew on every replica at about the same rate (~450 MB/hour). This was
true even for replicas with almost no local work. Every replica receives the
remote commit events of all the others, so the growth followed the cluster-wide
commit rate, not each replica's own load.
# Containers were repeatedly OOMKilled (exit code 137) while the Java heap was
far below its maximum.
* Example: container RSS reached 24.3 GB (the limit) while the Java heap was
11.4 GB, out of a 17.2 GB maximum.
* Increasing the container memory or tuning GC settings did not help.
# Some replicas were restarted by their liveness probe after long JVM pauses.
* This happened when a large backlog of ClassLoaderData was finally processed.
* It also happened when we ran diagnostic commands that walk all class loaders
(jcmd VM.classloader_stats).
h2. Evidence
h3. 1. NMT summary of a replica after ~11 hours
(-XX:NativeMemoryTracking=summary)
{code:java}
Total: reserved=26540MB, committed=12155MB
malloc: 7024MB #31316799
- Java Heap (reserved=17216MB, committed=4624MB)
- Class (reserved=4439MB, committed=3434MB)
(classes #28771)
(malloc=3415MB #12733544) (peak=4961MB #18457246)
( Metadata: )
( reserved=192MB, committed=142MB)
( used=140MB)
- Module (reserved=2545MB, committed=2545MB)
(malloc=2545MB #7582516) (peak=3697MB #11015426)
- Synchronization (reserved=251MB, committed=251MB)
(malloc=251MB #2528005)
- Symbol (reserved=222MB, committed=222MB)
- Metaspace (reserved=193MB, committed=143MB)
{code}
* No class loading problem. The number of loaded classes (~28.8K) and
Metaspace (~143 MB) were stable.
* Growth was in malloc only. It was millions of small native allocations in
the Class, Module and Synchronization categories. Together they used more
memory than the committed Java heap.
* The allocation ratio points to ClassLoaderData. The ratio of allocations
(Class : Module : Synchronization ≈ 5 : 3 : 1) matches what the JVM allocates
for each registered class loader: dictionary, unnamed module and package
tables, and the metaspace lock.
h3. 2. The throw-away loaders are MultiClassLoader instances
About 5 minutes after a replica restart, under load:
$ jcmd <pid> VM.classloader_stats | grep -c MultiClassLoader 14091
That is about 47 new org.apache.openjpa.lib.util.MultiClassLoader loaders per
second.
h3. 3. Only a Full GC reclaims the memory
||Step||NMT Class (committed)||NMT Module (committed)||
|Fresh replica, before|36 MB|15 MB|
|jcmd <pid> GC.run (with -XX:+ExplicitGCInvokesConcurrent, so a G1 concurrent
cycle), +2 min|50 MB|26 MB|
|Some minutes later|72 MB|43 MB|
|jcmd <pid> GC.class_histogram (forces a Full GC)|17 MB|1 MB|
* After the Full GC, the histogram showed 0 live MultiClassLoader instances.
The loaders are unreachable garbage, not a reference leak.
* However, their native memory was not released by the G1 concurrent cycle
(Remark took place, as shown in the GC log). It was only released by the Full
GC.
* In a long-running service, Full GCs are rare, so the memory accumulates.
h3. 4. Standalone reproduction (no application code)
{code:java}
import java.io.*;
import java.util.*;
import org.apache.openjpa.util.Serialization;
public class Repro {
public static void main(String[] args) throws Exception {
ArrayList<Object> value = new ArrayList<>();
value.add(new HashMap<>(Map.of("k", 1L)));
value.add(new HashSet<>(Set.of("x")));
value.add(new Date());
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
new ObjectOutputStream(bytes).writeObject(value);
byte[] data = bytes.toByteArray();
for (int i = 0; i < 50_000; i++) {
new Serialization.ClassResolvingObjectInputStream(new
ByteArrayInputStream(data)).readObject();
}
Thread.sleep(5_000); // inspect with: jcmd <pid> VM.native_memory
summary / VM.classloader_stats
}
}
{code}
Run with {{java -XX:NativeMemoryTracking=summary -XX:+UseG1GC -Xmx512m -cp
openjpa-3.2.2.jar:.}} Repro. Results with JDK 17:
|| ||NMT Class malloc||NMT Module malloc||MultiClassLoader loaders registered||
|OpenJPA 3.2.2|168 MB (606,620 allocations)|125 MB (363,310 allocations)|92,308|
|With the patch below|0.1 MB|0.03 MB|0|
h2. Root cause
{{{}openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java{}}}:
* 3.2.2:
[https://github.com/apache/openjpa/blob/3.2.2/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java#L131-L141]
* master:
[https://github.com/apache/openjpa/blob/master/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java#L122-L131]
{code:java}
@Override
protected Class resolveClass(ObjectStreamClass desc)
throws IOException, ClassNotFoundException {
String name = BlacklistClassResolver.DEFAULT.check(desc.getName());
MultiClassLoader loader = AccessController
.doPrivileged(J2DoPrivHelper.newMultiClassLoaderAction()); // new
loader per class descriptor
addContextClassLoaders(loader);
loader.addClassLoader(getClass().getClassLoader());
loader.addClassLoader(MultiClassLoader.SYSTEM_LOADER);
return Class.forName(name, true, loader); // JVM
registers the new loader (native ClassLoaderData)
}
{code}
{{resolveClass()}} is called once for each class descriptor in the stream. A
single RemoteCommitEvent contains several class descriptors: the event class,
collections, and OpenJPA id types. So each received event registers several new
loaders with the JVM.
The hot caller is
{{{}TCPRemoteCommitProvider$TCPPortListener$ReceiveSocketHandler{}}}:
* 3.2.2:
[https://github.com/apache/openjpa/blob/3.2.2/openjpa-kernel/src/main/java/org/apache/openjpa/event/TCPRemoteCommitProvider.java#L712-L726]
* master:
[https://github.com/apache/openjpa/blob/master/openjpa-kernel/src/main/java/org/apache/openjpa/event/TCPRemoteCommitProvider.java#L704-L718]
{code:java}
ObjectInputStream ois = new Serialization.ClassResolvingObjectInputStream(in);
...
byte[] senderAddress = (byte[]) ois.readObject();
RemoteCommitEvent rce = (RemoteCommitEvent) ois.readObject();
Serialization.deserialize(...)
{code}
is also used by {{DBDictionary.getBlobObject()}} for serialized BLOB fields, so
applications that use such fields are affected in the same way.
On {{{}master{}}}, the {{doPrivileged}} wrapper was removed (OPENJPA-2940), but
{{resolveClass()}} still does {{new MultiClassLoader()}} per call. *The issue
is still present.*
h2. Fix we applied (tested in production-like load)
We changed only ClassResolvingObjectInputStream. The candidate loaders are
collected once per stream, in the same order as before. Classes are then
resolved directly against those existing loaders, so no new class loader is
ever passed to the JVM.
* The protected addContextClassLoaders(MultiClassLoader) hook is kept, so
PersistentObjectInputStream (which adds the StoreContext loader) works
unchanged.
* The MultiClassLoader instance is used only as an ordered, de-duplicating
container. It is never handed to Class.forName, so the JVM never registers it.
{code}
--- a/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java
(3.2.2)
+++ b/openjpa-kernel/src/main/java/org/apache/openjpa/util/Serialization.java
@@ public static class ClassResolvingObjectInputStream extends
ObjectInputStream {
+ /**
+ * Candidate loaders, in the same order the original MultiClassLoader
+ * used them; resolved lazily once per stream.
+ */
+ private ClassLoader[] _loaders;
+
public ClassResolvingObjectInputStream(InputStream delegate)
throws IOException {
super(delegate);
}
@Override
protected Class resolveClass(ObjectStreamClass desc)
throws IOException, ClassNotFoundException {
String name = BlacklistClassResolver.DEFAULT.check(desc.getName());
- MultiClassLoader loader = AccessController
- .doPrivileged(J2DoPrivHelper.newMultiClassLoaderAction());
- addContextClassLoaders(loader);
- loader.addClassLoader(getClass().getClassLoader());
- loader.addClassLoader(MultiClassLoader.SYSTEM_LOADER);
- return Class.forName(name, true, loader);
+
+ ClassNotFoundException notFound = null;
+ for (ClassLoader loader : getCandidateLoaders()) {
+ try {
+ return Class.forName(name, true, loader);
+ } catch (ClassNotFoundException e) {
+ if (notFound == null)
+ notFound = e;
+ }
+ }
+
+ // primitive types and anything the candidates could not see
+ try {
+ return super.resolveClass(desc);
+ } catch (ClassNotFoundException e) {
+ throw notFound != null ? notFound : e;
+ }
+ }
+
+ private ClassLoader[] getCandidateLoaders() {
+ if (_loaders == null) {
+ // MultiClassLoader is used here only as an ordered,
de-duplicating
+ // container so that subclasses overriding
addContextClassLoaders keep
+ // working; it is never handed to the JVM, hence no
ClassLoaderData
+ // is created for it.
+ MultiClassLoader holder = new MultiClassLoader();
+ addContextClassLoaders(holder);
+ holder.addClassLoader(getClass().getClassLoader());
+ holder.addClassLoader(MultiClassLoader.SYSTEM_LOADER);
+ _loaders = java.util.Arrays.stream(holder.getClassLoaders())
+ .filter(java.util.Objects::nonNull)
+ .distinct()
+ .toArray(ClassLoader[]::new);
+ }
+ return _loaders;
}
{code}
h3. Notes on behavior
* Lookup order is unchanged. The order is: thread context class loader, then
(for PersistentObjectInputStream) the StoreContext loader, then the stream's
own class loader, then the system class loader. Bootstrap classes are found
through normal parent delegation of each candidate.
* Initialization is unchanged. initialize = true is kept, as in the original.
* Added fallback. If no candidate can load the class, we fall back to
ObjectInputStream.resolveClass(desc), which also handles primitive types. If
that also fails, the first ClassNotFoundException is rethrown.
* Loaders are fixed per stream. The candidate list is computed once per stream
instance, so the context class loader is captured on the first resolveClass
call. For TCPRemoteCommitProvider the stream lives on a single receiver thread,
so this is equivalent to the original behavior.
h3. Result
* The reproduction above no longer creates any class loader (see the table in
Evidence 4).
* In a repeated 15+ hour load test on the same 6-replica cluster, the native
memory growth and the OOMKilled restarts no longer occur.
h3. Possible alternatives for the upstream fix
* Cache a single MultiClassLoader per stream (or per set of loaders) and reuse
it, instead of creating one per resolveClass() call. This still registers one
loader per stream, so it is less effective than resolving directly against the
candidate loaders.
* Review other places where a MultiClassLoader is created per call and then
passed to Class.forName. OPENJPA-2057 notes that class-loading decisions are
spread across the code base.
----
h2. Related existing issues (no exact duplicate found)
Searched on 2026-10-07 for MultiClassLoader, ClassResolvingObjectInputStream,
TCPRemoteCommitProvider, classloader leak, native memory, Metaspace and
ClassLoaderData.
||Issue||Status||Relation||
|OPENJPA-2454 Memory leak when using multiple classloaders|Open (2.2.2)|Also a
class-loader-related memory leak, but caused by application-created loaders
retaining classes. Different mechanism.|
|OPENJPA-2057 Rethinking ClassLoading architecture|Open|Notes that
class-loading strategy is spread across many places, including ad-hoc
MultiClassLoader usage. This report is one concrete runtime consequence.|
|OPENJPA-1755 MultiClassLoader relies on its underlying ClassLoaders for
hashCode|Open|Unrelated defect in MultiClassLoader. Its discussion assumes
MultiClassLoaders are not created in large numbers, which is not true on this
code path.|
|OPENJPA-2801 Kubernetes TCPRemoteCommitProvider|Resolved|Introduced the
provider we use. The leak is in the shared TCP receive path, so it affects all
TCPRemoteCommitProvider subclasses.|
|OPENJPA-2538, OPENJPA-2636, OPENJPA-285|Closed|Earlier class loader leaks
(PCRegistry, custom plugins). Different code paths.|
----
h2.
Best regards,
Greta Jang
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
