[ http://issues.apache.org/jira/browse/LUCENE-488?page=comments#action_12363568 ]
Daniel Naber commented on LUCENE-488: ------------------------------------- writer.setMaxBufferedDocs(1) was a bad idea, it doesn't work because of an off-by-one bug. writer.setMaxBufferedDocs(2) should work, but I had to stop the unit test because it's too slow because of the many disk accesses. Other things to try: -get stack trace of OOM (requires java 1.5) -use writer.setUseCompoundFile(false) and look at the index directory after the crash -use writer.setInfoStream(System.out) to get some (not much) more output from Lucene BTW, this seems to affect all big stored fields, not just binary fields. (Please reply here in the issue tracker, not on the mailing list. This way things can be properly tracked). > adding docs with large (binary) fields of 5mb causes OOM regardless of heap > size > -------------------------------------------------------------------------------- > > Key: LUCENE-488 > URL: http://issues.apache.org/jira/browse/LUCENE-488 > Project: Lucene - Java > Type: Bug > Versions: 1.9 > Environment: Linux asimov 2.6.6.hoss1 #1 SMP Tue Jul 6 16:31:01 PDT 2004 > i686 GNU/Linux > Reporter: Hoss Man > Attachments: TestBigBinary.java > > as reported by George Washington in a message to [email protected] > with subect "Storing large text or binary source documents in the index and > memory usage" arround 2006-01-21 there seems to be a problem with adding docs > containing really large fields. > I'll attach a test case in a moment, note that (for me) regardless of how big > i make my heap size, and regardless of what value I set MIN_MB to, once it > starts trying to make documents of containing 5mb of data, it can only add 9 > before it rolls over and dies. > here's the output from the code as i will attach in a moment... > [junit] Testsuite: org.apache.lucene.document.TestBigBinary > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 78.656 sec > [junit] ------------- Standard Output --------------- > [junit] NOTE: directory will not be cleaned up automatically... > [junit] Dir: > /tmp/org.apache.lucene.document.TestBigBinary.97856146.100iters.4mb > [junit] iters completed: 100 > [junit] totalBytes Allocated: 419430400 > [junit] NOTE: directory will not be cleaned up automatically... > [junit] Dir: > /tmp/org.apache.lucene.document.TestBigBinary.97856146.100iters.5mb > [junit] iters completed: 9 > [junit] totalBytes Allocated: 52428800 > [junit] ------------- ---------------- --------------- > [junit] Testcase: > testBigBinaryFields(org.apache.lucene.document.TestBigBinary): Caused an > ERROR > [junit] Java heap space > [junit] java.lang.OutOfMemoryError: Java heap space > [junit] Test org.apache.lucene.document.TestBigBinary FAILED -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
