[
https://issues.apache.org/jira/browse/LUCENE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doron Cohen resolved LUCENE-488.
--------------------------------
Resolution: Fixed
This problem was resolved by LUCENE-843, after which stored fields are written
directly into the directory (therefore not consuming aggregated RAM).
It is interesting that the test provided here was allocating a new byte buffer
of 2 - 10 MB for each added doc. This by itself couldeventually lead to OOMs
because as the program ran longer it was becoming harder to alocate consecutive
chunks of those sizes. Enhancing binary fields with offset and length (?)
would allow applications to reuse the input byte array and allocate less of
those.
> adding docs with large (binary) fields of 5mb causes OOM regardless of heap
> size
> --------------------------------------------------------------------------------
>
> Key: LUCENE-488
> URL: https://issues.apache.org/jira/browse/LUCENE-488
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 1.9
> Environment: Linux asimov 2.6.6.hoss1 #1 SMP Tue Jul 6 16:31:01 PDT
> 2004 i686 GNU/Linux
> Reporter: Hoss Man
> Attachments: TestBigBinary.java
>
>
> as reported by George Washington in a message to [EMAIL PROTECTED] with
> subect "Storing large text or binary source documents in the index and memory
> usage" arround 2006-01-21 there seems to be a problem with adding docs
> containing really large fields.
> I'll attach a test case in a moment, note that (for me) regardless of how big
> i make my heap size, and regardless of what value I set MIN_MB to, once it
> starts trying to make documents of containing 5mb of data, it can only add 9
> before it rolls over and dies.
> here's the output from the code as i will attach in a moment...
> [junit] Testsuite: org.apache.lucene.document.TestBigBinary
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 78.656 sec
> [junit] ------------- Standard Output ---------------
> [junit] NOTE: directory will not be cleaned up automatically...
> [junit] Dir:
> /tmp/org.apache.lucene.document.TestBigBinary.97856146.100iters.4mb
> [junit] iters completed: 100
> [junit] totalBytes Allocated: 419430400
> [junit] NOTE: directory will not be cleaned up automatically...
> [junit] Dir:
> /tmp/org.apache.lucene.document.TestBigBinary.97856146.100iters.5mb
> [junit] iters completed: 9
> [junit] totalBytes Allocated: 52428800
> [junit] ------------- ---------------- ---------------
> [junit] Testcase:
> testBigBinaryFields(org.apache.lucene.document.TestBigBinary): Caused an
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Test org.apache.lucene.document.TestBigBinary FAILED
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
