[
https://issues.apache.org/jira/browse/LUCENE-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118723#comment-13118723
]
Uwe Schindler commented on LUCENE-3463:
---------------------------------------
Some further issue hunting showed the followinbg problems with the current
patch, that make it almost impossible to implement correctly without
redesigning RandomCodecProvider in LTC:
- Mike have seen problems with running all tests and suddenly got errors about
missing "SimpleText" or "Memory" codec. The reason is simple: once a field uses
a codec this is preserved to infinity in the previousMappings map. This problem
could be solved by clearing the map before each test class/test method. The
field->codec mappings are not random, they are based on a a final per
codec-provider instance seed and the field's hashcode and modulo the size of
codecs.
- My patch violates the reproducibility, as unregistering and readding a codec
from the RandomCodecProvider changes the order in the list and modifies the
selection of codecs based on field name. Theoretically, RandomCodecProvider
should not allow any modifications to its registered codecs after creating (and
the swapCodecs operations in LTC).
A possible solution might be to have two RCPs created by LTC and only
exchanging them based on annotation. Problem is that both still have separate
mappings from field name to codec -> does not work if you reuse indexes across
test methods. Another possible solution might be to enable a lowMemory mode for
RCP (via a setter, I already implemented that somehow), so it just never
returns memory/simpletext (maybe replace by standard codec). Problem here
again, the mapping from field name to codec is preserved for all codecs, but if
a mapping to a simpletext/memory codec already exists for field name, it cannot
be used (so the fieldname is "dead" for this test. This is also a no-go.
I think we should move the annotation, as Robert suggests, up to the class, so
all methods in a class will get the reduced set of codecs. Before each class
(beforeClass) we cleanup the codec provider and recreate a new one according to
the annotation. The easy way is then (as we did for some other tests, too), to
move the tests that should not run on simpletext/memory to separate classes -
this is the easiest way to go.
> Jenkins trunk tests (nightly only) fail quite often with OOM in Automaton/FST
> tests
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-3463
> URL: https://issues.apache.org/jira/browse/LUCENE-3463
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3463.patch, LUCENE-3463.patch, LUCENE-3463.patch,
> LUCENE-3463.patch
>
>
> The nightly Job Lucene-trunk quite often fails with OOM (in several methods,
> not always in the same test):
> Example from last night (this time a huge Automaton):
> {noformat}
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Dumping heap to
> /home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
> ...
> [junit] Heap dump file created [86965954 bytes in 1.186 secs]
> [junit] Testsuite: org.apache.lucene.index.TestTermsEnum
> [junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum):
> Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] at
> org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:128)
> [junit] at
> org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:28)
> [junit] at
> org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:134)
> [junit] at
> org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
> [junit] at
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit]
> [junit]
> [junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
> {noformat}
> Other traces:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
> [junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs): Caused
> an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> [junit] at
> org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
> [junit] at
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
> [junit] at
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
> [junit] at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
> [junit] at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> [junit] at
> org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> [junit] at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
> [junit] at
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
> [junit] at
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
> [junit] at
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
> [junit] at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> [junit] at
> org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> [junit] at
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
> [junit] at
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
> [junit] at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
> [junit] at
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
> [junit] at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> [junit] at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> [junit] at
> org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> [junit] at
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> {noformat}
> or:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
> [junit] Testcase:
> testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton): Caused an
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] at
> org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:128)
> [junit] at
> org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:28)
> [junit] at
> org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:134)
> [junit] at
> org.apache.lucene.util.automaton.TestCompiledAutomaton.build(TestCompiledAutomaton.java:39)
> [junit] at
> org.apache.lucene.util.automaton.TestCompiledAutomaton.testTerms(TestCompiledAutomaton.java:55)
> [junit] at
> org.apache.lucene.util.automaton.TestCompiledAutomaton.testRandom(TestCompiledAutomaton.java:101)
> [junit] at
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> {noformat}
> Almost every nightly test fails, history:
> [https://builds.apache.org/job/Lucene-trunk]
> We should maybe raise the max heap space or reduce doc counts/...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
