[
https://issues.apache.org/jira/browse/TIKA-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490821#comment-15490821
]
Luis Filipe Nassif commented on TIKA-2058:
------------------------------------------
Hi [~gagravarr],
The idea of the patch is exactly to not record read-only buffers that do not
need special unmapping. There was one instance of FileBackedDataSource
consuming 1.5GB of heap, and I think that is a large amount of heap. The
remaining 4.3GB of HeapByteBuffers were not clear to me from where they are,
but I searched the source of many of them individually and all was referenced
by a FileBackedDataSource. Maybe jvisualvm could compute retained sizes wrongly?
If the problematic file was discovered (if it exists) that would help a lot...
> Memory Leak in Tika version 1.13 when parsing millions of files
> ---------------------------------------------------------------
>
> Key: TIKA-2058
> URL: https://issues.apache.org/jira/browse/TIKA-2058
> Project: Tika
> Issue Type: Bug
> Affects Versions: 1.13
> Reporter: Tim Barrett
> Attachments: poi-3.15-beta1-p1.jar, poi-3.15-beta1-p1.pom,
> prevents-OOM-when-writable-is-false.patch, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We have an application using Tika which parses roughly 7,000,000 files of
> different types, many of the files are MSG files with attachments. This works
> correctly with Tika 1.9, and has been in production for over a year, with
> parsing runs taking place every few weeks. The same application runs into
> insufficient memory problems (java heap) when using Tika 1.13.
> I have used lsof and file leak detector to track down open files, however
> neither shows any open files when the application is running. I did find an
> issue with open files https://issues.apache.org/jira/browse/TIKA-2015,
> however there was a workaround for this and this is not the issue.
> I am sorry to have to report this with a level of vagueness, but with lsof
> turning nothing up I am a bit stuck as to how to investigate further. We are
> more than willing to help by testing on the basis of any ideas provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
