This is an automated email from the ASF dual-hosted git repository. tballison pushed a commit to branch TIKA-4835-spill-sites in repository https://gitbox.apache.org/repos/asf/tika.git
commit a2eabc69437109a15d85ce8c81315380d44eb277 Merge: 4a687bbe54 ab22ec9c9a Author: tallison <[email protected]> AuthorDate: Wed Aug 26 18:02:33 2026 -0400 Merge remote-tracking branch 'origin/main' into TIKA-4835-spill-sites CHANGES.txt | 18 ++++ .../org/apache/tika/pipes/core/PipesClient.java | 22 +++-- .../org/apache/tika/pipes/core/PipesParser.java | 9 ++ .../org/apache/tika/pipes/core/ServerManager.java | 23 +++++ .../tika/pipes/core/SharedServerManager.java | 71 ++++++++++++++-- .../pipes/core/SharedServerManagerClosedTest.java | 45 ++++++++++ .../tika/pipes/core/SharedServerModeTest.java | 98 ++++++++++++++++++++++ 7 files changed, 275 insertions(+), 11 deletions(-) diff --cc CHANGES.txt index e22631edfa,7c0d7b9084..195ea6035d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,27 -1,23 +1,45 @@@ Release 4.1.0 - unreleased + * Parsers and detectors no longer spool in-memory input to a temp file just + to read it back with random access: the JPEG, TIFF and WebP parsers, the + OpenDocument parser's inline pictures, the OLE2 container detector's + entry-name read, and PDFParser (the incremental-update scan and the main + load). Together these were the bulk of the temp bytes behind 4.0.0's + batch slowdown on spinning disks (see docs/.../pipes/performance.adoc). + One rule everywhere: content goes through the stream cache (bounded by + the pipes cache memory budget, 1MB per object without one); what stays + in memory is read in place through a zero-copy view, what spills is + read from the file. PDFBox and metadata-extractor no longer receive a + second heap copy of the document. The PDF renderer now re-opens the + document from byte 0 for every render, which fixes per-page rendering + (RENDER_PAGES_AT_PAGE_END) silently producing only the first page -- + for file-backed input too. PDFParser.getPDDocumentFromStream is + deprecated and no longer called; override the new + getPDDocument(RandomAccessRead, ...) instead. ImageMetadataExtractor + gains TikaInputStream and InputStream overloads. One visible metadata + change: JPEG/TIFF/WebP images read from memory no longer carry + metadata-extractor's file-system tags (img:File Name, img:File Size, + img:File Modified Date), which described whatever file was read rather + than the image; input backed by a file still carries them (TIKA-4835). + + * Shared pipes server (useSharedServer: true, not the default): a client whose + in-flight parse was killed by another client's restart could restart the + healthy replacement. ensureRunning holds its lock across the whole fork, so + siblings cannot report a dead worker until after the replacement is up, and + the pending-restart flag carried no process identity -- so a report about + the process that just died was applied to its successor, which was then + destroyed and re-forked. One worker death produced two restarts and a second + round of destroyed in-flight work; under sustained concurrent load it + sustained itself at one spurious restart per round, appearing as periodic + unexplained worker churn and intermittent parse failures that succeed on + retry. Each fork now carries a generation that clients capture when they + connect and hand back with every report, and reports about a superseded + process are dropped. Also fixed in shared mode: ensureRunning could fork a + replacement after shutdown() that nothing owned and nothing would ever + destroy, and an interrupt during process teardown left the process handle + pointing at a killed process and leaked the temp directory. Affects 4.0.0 + and earlier (TIKA-4844). + * tika-pipes: the cache memory budget (how much rewindable content a forked worker keeps in memory before spilling to disk; new since 4.0.0, which had no budget at all) defaults to a quarter of the fork's heap, so raising
