[
https://issues.apache.org/jira/browse/XMLBEANS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123104#comment-18123104
]
PJ Fanning commented on XMLBEANS-676:
-------------------------------------
I suspect the issues could be more related to changes in ZipArchiveFakeEntry in
POI.
The changes were suggested by AI to improve protections against corrupt files
but have already led to me having to change the settings to poi-integration
memory setup.
// this might help your test harness
ZipInputStreamZipEntrySource.setThresholdBytesForTempFiles(2_000_000);
Recent zip-entry changes that increased heap use (seen in poi-integration)
1. #1236 (76d93fcc0f), the main cause. Entries with an unknown size (getSize()
== -1, i.e. data-descriptor entries such as the ones POI itself writes to a
stream) used to go straight to a temp file whenever
ZipInputStreamZipEntrySource.setThresholdBytesForTempFiles was set. Now they
are kept in memory up to the full threshold and only spilled to disk beyond it.
ZipInputStreamZipEntrySource keeps every entry until the package is closed, so
with a 16 MB threshold each such part of a stream-opened package can now take
up to 16 MB of heap where it used to take almost none. Reading one also
allocates about 2× its size briefly (the doubling chunks of
UnsynchronizedByteArrayOutputStream plus the toByteArray() copy).
2. #1227 (a6e29606b7) → #1235 (8e06374bd0). The fix for eager allocation from
untrusted entry sizes now limits the first buffer for known-size entries to 2
MB (MAX_INIT_BUFFER_SIZE). Larger entries then grow through commons-io's
chunked buffer (2 + 4 + 8 + … MB) and finish with a full toByteArray() copy.
That is roughly 24 MB of allocation for a 10 MB entry where it used to be 20
MB, plus many more short-lived large arrays. (#1227 alone was worse, since it
read everything through a 4 KB buffer that doubled; #1235 brought back sized
reads up to 2 MB.)
3. #1277 (187b58519d) hid #1236 rather than fixing it. It cut TestAllFiles'
temp-file threshold from 16 MB to 2 MB and raised the poi-integration heap to 3
GB. The tests passed again, but any user who sets a temp-file threshold still
gets the #1236 behaviour. (#1277 did also bring real improvements: a direct
read into the result array for sized reads ≤ 2 MB, and the handlers now close
stream-opened OPCPackages.)
I am experimenting with changes that allow us to keep some of the benefits of
the changes above but that reduce memory usage.
> ThreadLocals in NamespaceContext are sometimes left over and cause a memory
> leak
> --------------------------------------------------------------------------------
>
> Key: XMLBEANS-676
> URL: https://issues.apache.org/jira/browse/XMLBEANS-676
> Project: XMLBeans
> Issue Type: Bug
> Affects Versions: 5.4.1
> Reporter: Dominik Stadler
> Priority: Major
> Attachments: image-2026-10-04-10-19-21-146.png,
> image-2026-10-04-10-20-36-513.png
>
>
> When running the large regression test-suite for Apache POI, I saw that over
> time, more and more memory is allocated by thread-locals in NamespaceContext
> and is not freed any more.
> The test-application reads millions of documents in multiple threads. It
> re-uses the threads for multiple documents.
> Looking at memory dumps, it seems the thread-local in NamespaceContext keeps
> considerable amounts of memory from being freed in at least two threads.
> It seems sometimes the pushed elements are not pop()ed from the stack
> properly.
> A quick look at the code did not show how this can happen, it seems calls to
> "push()" are always followed by a "pop()" in a try-finally block. So elements
> should always be removed again.
> Also pop()ing the last item should always clear the thread-local, but one of
> the two threads has an empty stack with "current" still holding onto a large
> portion of memory.
> The application is using Thread.stop() in some cases, but this should cause
> the thread to be re-created by the executor and thus thread-locals to be
> freed. The application also triggers all sorts of exceptions, including OOMs
> as part of processing files.
>
> As a result, the regression testing application now runs very slowly as it
> constantly causes OOMs as lots of memory is used by the thread-locals.
>
> Thread holding onto memory in "current" inside the thread-local:
> !image-2026-10-04-10-19-21-146.png!
>
> Thread holding onto memory via a large number of elements remaining in the
> stack:
> !image-2026-10-04-10-20-36-513.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
