>>But... how come setting IW's RAM buffer doesn't prevent the OOMs? I've been setting the IndexWriter RAM buffer to 300 meg and giving the JVM 1gig.
Last run I gave the JVM 3 gig, with writer settings of RAM buffer=300 meg, merge factor=20, term interval=8192, usecompound=false. All fields are ANALYZED_NO_NORMS. Lucene version is a 2.9 build, JVM is Sun 64bit 1.6.0_07. This graphic shows timings for 100 consecutive write sessions, each adding 30,000 documents, committing and then closing : http://tinyurl.com/anzcjw You can see the periodic merge costs and then a big spike towards the end before it crashed. The crash details are here after adding ~3 million documents in 98 write sessions: This batch index session added 3000 of 30000 docs : 10% complete Exception in thread "Thread-280" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Unknown Source) at java.lang.String..<init>(Unknown Source) at org.apache.lucene.search.trie.TrieUtils.longToPrefixCoded(TrieUtils.java:148) at org.apache.lucene.search.trie.TrieUtils.trieCodeLong(TrieUtils.java:302) at test.LongTrieAnalyzer$LongTrieTokenStream.next(LongTrieAnalyzer.java:49) at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:159) at org.apache.lucene.index.DocFieldConsumersPerField.processFields(DocFieldConsumersPerField.java:36) at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:234) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:762) at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:740) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2039) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2013) at test.IndexMarksFile$IndexingThread.run(IndexMarksFile.java:319) Exception in thread "Thread-281" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.commons.csv.CharBuffer.toString(CharBuffer.java:177) at org.apache.commons.csv.CSVParser.getLine(CSVParser.java:242) at test.IndexMarksFile.getLuceneDocument(IndexMarksFile.java:272) at test.IndexMarksFile$IndexingThread.run(IndexMarksFile.java:314) Committing Closing Exception in thread "main" java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3569) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3660) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3634) at test.IndexMarksFile.run(IndexMarksFile.java:176) at test.IndexMarksFile.main(IndexMarksFile.java:101) at test.MultiIndexAndRun.main(MultiIndexAndRun.java:49) For each write session I have a single writer, and 2 indexing threads adding documents through this writer. There are no updates/deletes - only adds. When both indexing threads complete the primary thread commits and closes the writer. I then open a searcher run some search benchmarks, close the searcher and start another write session. The documents have ~12 fields and are all the same size so I don't think this OOM is down to rogue data. Each field has 100 near-unique tokens. The files on disk after the crash are as follows: 1930004059 Mar 9 13:32 _106.fdt 2731084 Mar 9 13:32 _106.fdx 175 Mar 9 13:30 _106.fnm 1190042394 Mar 9 13:39 _106.frq 814748995 Mar 9 13:39 _106.prx 16512596 Mar 9 13:39 _106.tii 1151364311 Mar 9 13:39 _106.tis 1949444533 Mar 9 14:53 _139.fdt 2758580 Mar 9 14:53 _139.fdx 175 Mar 9 14:51 _139.fnm 1202044423 Mar 9 15:00 _139.frq 822954002 Mar 9 15:00 _139.prx 16629104 Mar 9 15:00 _139.tii 1159392207 Mar 9 15:00 _139.tis 1930102055 Mar 9 16:15 _16c.fdt 2731084 Mar 9 16:15 _16c.fdx 175 Mar 9 16:13 _16c.fnm 1190090014 Mar 9 16:22 _16c.frq 814763781 Mar 9 16:22 _16c.prx 16514967 Mar 9 16:22 _16c.tii 1151524173 Mar 9 16:22 _16c.tis 1928053697 Mar 9 17:52 _19e.fdt 2728260 Mar 9 17:52 _19e.fdx 175 Mar 9 17:46 _19e.fnm 1188837093 Mar 9 18:08 _19e.frq 813915820 Mar 9 18:08 _19e.prx 16501902 Mar 9 18:08 _19e.tii 1150623773 Mar 9 18:08 _19e.tis 1951474247 Mar 9 20:22 _1cj.fdt 2761396 Mar 9 20:22 _1cj.fdx 175 Mar 9 20:18 _1cj.fnm 1203285781 Mar 9 20:39 _1cj.frq 823797656 Mar 9 20:39 _1cj.prx 16639997 Mar 9 20:39 _1cj.tii 1160143978 Mar 9 20:39 _1cj.tis 1929978366 Mar 10 01:02 _1fm.fdt 2731060 Mar 10 01:02 _1fm.fdx 175 Mar 10 00:43 _1fm.fnm 1190031780 Mar 10 02:36 _1fm.frq 814741146 Mar 10 02:36 _1fm.prx 16513189 Mar 10 02:36 _1fm.tii 1151399139 Mar 10 02:36 _1fm.tis 189073186 Mar 10 01:51 _1ft.fdt 267556 Mar 10 01:51 _1ft.fdx 175 Mar 10 01:50 _1ft.fnm 110750150 Mar 10 02:04 _1ft.frq 79818488 Mar 10 02:04 _1ft.prx 2326691 Mar 10 02:04 _1ft.tii 165932844 Mar 10 02:04 _1ft.tis 212500024 Mar 10 03:16 _1g5.fdt 300684 Mar 10 03:16 _1g5.fdx 175 Mar 10 03:16 _1g5.fnm 125179984 Mar 10 03:28 _1g5.frq 89703062 Mar 10 03:28 _1g5.prx 2594360 Mar 10 03:28 _1g5.tii 184495760 Mar 10 03:28 _1g5.tis 64323505 Mar 10 04:09 _1gc.fdt 91020 Mar 10 04:09 _1gc.fdx 105283820 Mar 10 04:48 _1gf.fdt 148988 Mar 10 04:48 _1gf.fdx 175 Mar 10 04:09 _1gf.fnm 1491 Mar 10 04:09 _1gf.frq 4 Mar 10 04:09 _1gf.nrm 2388 Mar 10 04:09 _1gf.prx 254 Mar 10 04:09 _1gf.tii 15761 Mar 10 04:09 _1gf.tis 191035191 Mar 10 04:09 _1gg.fdt 270332 Mar 10 04:09 _1gg.fdx 175 Mar 10 04:09 _1gg.fnm 111958741 Mar 10 04:24 _1gg.frq 80645411 Mar 10 04:24 _1gg.prx 2349153 Mar 10 04:24 _1gg.tii 167494232 Mar 10 04:24 _1gg.tis 175 Mar 10 04:20 _1gh.fnm 10223275 Mar 10 04:20 _1gh.frq 4 Mar 10 04:20 _1gh.nrm 9056546 Mar 10 04:20 _1gh.prx 329012 Mar 10 04:20 _1gh.tii 23846511 Mar 10 04:20 _1gh.tis 175 Mar 10 04:28 _1gi.fnm 10221888 Mar 10 04:28 _1gi.frq 4 Mar 10 04:28 _1gi.nrm 9054280 Mar 10 04:28 _1gi.prx 328980 Mar 10 04:28 _1gi.tii 23843209 Mar 10 04:28 _1gi.tis 175 Mar 10 04:35 _1gj.fnm 10222776 Mar 10 04:35 _1gj.frq 4 Mar 10 04:35 _1gj.nrm 9054943 Mar 10 04:35 _1gj.prx 329060 Mar 10 04:35 _1gj.tii 23849395 Mar 10 04:35 _1gj.tis 175 Mar 10 04:42 _1gk.fnm 10220381 Mar 10 04:42 _1gk.frq 4 Mar 10 04:42 _1gk.nrm 9052810 Mar 10 04:42 _1gk.prx 329029 Mar 10 04:42 _1gk.tii 23845373 Mar 10 04:42 _1gk.tis 175 Mar 10 04:48 _1gl.fnm 9274170 Mar 10 04:48 _1gl.frq 4 Mar 10 04:48 _1gl.nrm 8226681 Mar 10 04:48 _1gl.prx 303327 Mar 10 04:48 _1gl.tii 21996826 Mar 10 04:48 _1gl.tis 22418126 Mar 10 04:58 _1gm.fdt 31732 Mar 10 04:58 _1gm.fdx 175 Mar 10 04:57 _1gm.fnm 10216672 Mar 10 04:57 _1gm.frq 4 Mar 10 04:57 _1gm.nrm 9049487 Mar 10 04:57 _1gm.prx 328813 Mar 10 04:57 _1gm.tii 23829627 Mar 10 04:57 _1gm.tis 175 Mar 10 04:58 _1gn.fnm 392014 Mar 10 04:58 _1gn.frq 4 Mar 10 04:58 _1gn.nrm 415225 Mar 10 04:58 _1gn.prx 24695 Mar 10 04:58 _1gn.tii 1816750 Mar 10 04:58 _1gn.tis 683 Mar 10 04:58 segments_7t 20 Mar 10 04:58 segments.gen 1935727800 Mar 9 11:17 _u1.fdt 2739180 Mar 9 11:17 _u1.fdx 175 Mar 9 11:15 _u1.fnm 1193583522 Mar 9 11:25 _u1.frq 817164507 Mar 9 11:25 _u1.prx 16547464 Mar 9 11:25 _u1..tii 1153764013 Mar 9 11:25 _u1.tis 1949493315 Mar 9 12:21 _x3.fdt 2758580 Mar 9 12:21 _x3.fdx 175 Mar 9 12:18 _x3.fnm 1202068425 Mar 9 12:29 _x3.frq 822963200 Mar 9 12:29 _x3.prx 16629485 Mar 9 12:29 _x3.tii 1159419149 Mar 9 12:29 _x3.tis Any ideas? I'm out of settings to tweak here. Cheers, Mark ----- Original Message ---- From: Michael McCandless <luc...@mikemccandless.com> To: java-user@lucene.apache.org Sent: Tuesday, 10 March, 2009 0:01:30 Subject: Re: A model for predicting indexing memory costs? mark harwood wrote: > > I've been building a large index (hundreds of millions) with mainly > structured data which consists of several fields with mostly unique values. > I've been hitting out of memory issues when doing periodic commits/closes > which I suspect is down to the sheer number of terms. > > I set the IndexWriter..setTermIndexInterval to 8 times the normal size of 128 > (an intervalof 1024) which delayed the onset of the issue but still failed. I think that setting won't change how much RAM is used when writing. > I'd like to get a little more scientific about what to set here rather than > simply experimenting with settings and hoping it doesn't fail again. > > Does anyone have a decent model worked out for how much memory is consumed at > peak? I'm guessing the contributing factors are: > > * Numbers of fields > * Numbers of unique terms per field > * Numbers of segments? Number of net unique terms (across all fields) is a big driver, but also net number of term occurrences, and how many docs. Lots of tiny docs take more RAM than fewer large docs, when # occurrences are equal. But... how come setting IW's RAM buffer doesn't prevent the OOMs? IW should simply flush when it's used that much RAM. I don't think number of segments is a factor. Though mergeFactor is, since during merging the SegmentMerger holds SegmentReaders open, and int[] maps (if there are any deletes) for each segment. Do you have a large merge taking place when you hit the OOMs? Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org