> I get really belligerent when being told to solve problems while wearing a > ball-and-chain.
I seem to have touched quite a nerve there then, Erick ;) I appreciate your sympathy. To be fair I haven't exhausted all possible avenues in changing the environment but I do remain interested in understanding more about memory consumption. I think a modelling/sizing spreadsheetwould be a useful addition to our documentation. ----- Original Message ---- From: Erick Erickson <erickerick...@gmail.com> To: java-user@lucene.apache.org Sent: Tuesday, 10 March, 2009 15:08:24 Subject: Re: A model for predicting indexing memory costs? You have my sympathy. Let's see, you're being told "we can't give you the tools you need to diagnose/fix the problem, but fix it anyway". Probably with the addendum "And fix it by Friday". You might want to consider staging a mutiny until "the powers that be" can give you a solution. Perhaps working with the system admins to set up profiling. Perhaps a temporary ID that has enough privileges to do what you need that can then be deleted. Perhaps....... In case you're wondering, I've been in that situation often enough that I get really belligerent when being told to solve problems while wearing a ball-and-chain. Invariably it takes more company time/resources than getting what I need to go forward directly. I mean, you'll be in a situation where you'll only be able to say "Well, I changed some things. Whether they were the right things I really can't say because I don't understand the root of the problem because you silly people won't let me use a profiler". And, even worse, if the problem goes away after you change something, you won't be able to say very much about whether or not it'll come back since you really don't know whether you've actually fixed anything or just masked the problem temporarily with the possibility that it'll come back when you add document N+1. By which time you (and others) will have to reconstruct all you know now, which is expensive. So, be much more diplomatic than this note, please <G>. But you may want to point out that without appropriate tools, your company may well spend significant time, yours and others, repeatedly trying to fix this issue. Through n+1 rounds. I've actually had good results by pointing out that it's not only *your* time that's at risk, but customers' time too. Whether you define customers as internal or external is irrelevant. Every round of diagnosis/fix carries the risk that N people waste time (and get paid for it). All to avoid a little up-front costs due to admin privileges (in this case). OK, enough ranting.. Best Erick On Tue, Mar 10, 2009 at 10:50 AM, mark harwood <markharw...@yahoo.co.uk>wrote: > > >>Could you get a heap dump (eg with YourKit) of what's using up all the > memory when you hit OOM? > > On this particular machine I have a JRE, no admin rights and therefore > limited profiling capability :( > That's why I was trying to come up with some formula for estimating memory > usage. > > >>When you say "write session", are you closing & opening a new IndexWriter > each time? > > Yes, commit then close. > > >>It seems likely this has something to do with merging, > > Presumably (norms + deleted arrays discounted) RAM usage for merges is not > proportional to number of terms or docs? I imagine the structures being > merged are streamed rather than loaded in whole or as a fixed percentage of > the whole. > > > > ----- Original Message ---- > From: Michael McCandless <luc...@mikemccandless.com> > To: java-user@lucene.apache.org > Sent: Tuesday, 10 March, 2009 14:23:59 > Subject: Re: A model for predicting indexing memory costs? > > > Mark, > > Could you get a heap dump (eg with YourKit) of what's using up all the > memory when you hit OOM? > > Also, can you turn on infoStream and post the output leading up to the OOM? > > When you say "write session", are you closing & opening a new IndexWriter > each time? Or, just calling .commit() and then re-using the same writer? > > It seems likely this has something to do with merging, though from your > listing I count 14 segments which shouldn't have been doing any merging at > mergeFactor=20, so that's confusing. > > Mike > > mark harwood wrote: > > > > >>> But.... how come setting IW's RAM buffer doesn't prevent the OOMs? > > > > I've been setting the IndexWriter RAM buffer to 300 meg and giving the > JVM 1gig. > > > > Last run I gave the JVM 3 gig, with writer settings of RAM buffer=300 > meg, merge factor=20, term interval=8192, usecompound=false. All fields are > ANALYZED_NO_NORMS. > > Lucene version is a 2.9 build, JVM is Sun 64bit 1.6.0_07. > > > > This graphic shows timings for 100 consecutive write sessions, each > adding 30,000 documents, committing and then closing : > > http://tinyurl.com/anzcjw > > You can see the periodic merge costs and then a big spike towards the end > before it crashed. > > > > The crash details are here after adding ~3 million documents in 98 write > sessions: > > > > This batch index session added 3000 of 30000 docs : 10% complete > > Exception in thread "Thread-280" java.lang.OutOfMemoryError: GC overhead > limit exceeded > > at java.util..Arrays.copyOf(Unknown Source) > > at java.lang.String...<init>(Unknown Source) > > at > org.apache.lucene.search.trie.TrieUtils.longToPrefixCoded(TrieUtils.java:148) > > at > org.apache.lucene.search.trie.TrieUtils.trieCodeLong(TrieUtils.java:302) > > at > test.LongTrieAnalyzer$LongTrieTokenStream.next(LongTrieAnalyzer.java:49) > > at > org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:159) > > at > org.apache.lucene.index.DocFieldConsumersPerField.processFields(DocFieldConsumersPerField.java:36) > > at > org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:234) > > at > org.apache.lucene.index..DocumentsWriter.updateDocument(DocumentsWriter.java:762) > > at > org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter..java:740) > > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2039) > > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2013) > > at test.IndexMarksFile$IndexingThread.run(IndexMarksFile.java:319) > > Exception in thread "Thread-281" java.lang.OutOfMemoryError: GC overhead > limit exceeded > > at org.apache.commons.csv.CharBuffer.toString(CharBuffer.java:177) > > at org.apache.commons.csv.CSVParser.getLine(CSVParser.java:242) > > at test.IndexMarksFile.getLuceneDocument(IndexMarksFile.java:272) > > at test.IndexMarksFile$IndexingThread.run(IndexMarksFile.java:314) > > Committing > > Closing > > Exception in thread "main" java.lang.IllegalStateException: this writer > hit an OutOfMemoryError; cannot commit > > at > org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3569) > > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3660) > > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3634) > > at test.IndexMarksFile.run(IndexMarksFile.java:176) > > at test..IndexMarksFile.main(IndexMarksFile.java:101) > > at test.MultiIndexAndRun.main(MultiIndexAndRun.java:49) > > > > > > For each write session I have a single writer, and 2 indexing threads > adding documents through this writer. There are no updates/deletes - only > adds. When both indexing threads complete the primary thread commits and > closes the writer.. > > I then open a searcher run some search benchmarks, close the searcher and > start another write session. > > The documents have ~12 fields and are all the same size so I don't think > this OOM is down to rogue data. Each field has 100 near-unique tokens. > > > > The files on disk after the crash are as follows: > > 1930004059 Mar 9 13:32 _106.fdt > > 2731084 Mar 9 13:32 _106.fdx > > 175 Mar 9 13:30 _106.fnm > > 1190042394 Mar 9 13:39 _106.frq > > 814748995 Mar 9 13:39 _106.prx > > 16512596 Mar 9 13:39 _106.tii > > 1151364311 Mar 9 13:39 _106.tis > > 1949444533 Mar 9 14:53 _139.fdt > > 2758580 Mar 9 14:53 _139.fdx > > 175 Mar 9 14:51 _139.fnm > > 1202044423 Mar 9 15:00 _139.frq > > 822954002 Mar 9 15:00 _139.prx > > 16629104 Mar 9 15:00 _139.tii > > 1159392207 Mar 9 15:00 _139.tis > > 1930102055 Mar 9 16:15 _16c.fdt > > 2731084 Mar 9 16:15 _16c.fdx > > 175 Mar 9 16:13 _16c.fnm > > 1190090014 Mar 9 16:22 _16c.frq > > 814763781 Mar 9 16:22 _16c.prx > > 16514967 Mar 9 16:22 _16c.tii > > 1151524173 Mar 9 16:22 _16c.tis > > 1928053697 Mar 9 17:52 _19e.fdt > > 2728260 Mar 9 17:52 _19e.fdx > > 175 Mar 9 17:46 _19e.fnm > > 1188837093 Mar 9 18:08 _19e.frq > > 813915820 Mar 9 18:08 _19e.prx > > 16501902 Mar 9 18:08 _19e.tii > > 1150623773 Mar 9 18:08 _19e.tis > > 1951474247 Mar 9 20:22 _1cj.fdt > > 2761396 Mar 9 20:22 _1cj.fdx > > 175 Mar 9 20:18 _1cj.fnm > > 1203285781 Mar 9 20:39 _1cj.frq > > 823797656 Mar 9 20:39 _1cj.prx > > 16639997 Mar 9 20:39 _1cj.tii > > 1160143978 Mar 9 20:39 _1cj.tis > > 1929978366 Mar 10 01:02 _1fm.fdt > > 2731060 Mar 10 01:02 _1fm.fdx > > 175 Mar 10 00:43 _1fm.fnm > > 1190031780 Mar 10 02:36 _1fm.frq > > 814741146 Mar 10 02:36 _1fm.prx > > 16513189 Mar 10 02:36 _1fm.tii > > 1151399139 Mar 10 02:36 _1fm.tis > > 189073186 Mar 10 01:51 _1ft.fdt > > 267556 Mar 10 01:51 _1ft.fdx > > 175 Mar 10 01:50 _1ft.fnm > > 110750150 Mar 10 02:04 _1ft.frq > > 79818488 Mar 10 02:04 _1ft.prx > > 2326691 Mar 10 02:04 _1ft.tii > > 165932844 Mar 10 02:04 _1ft.tis > > 212500024 Mar 10 03:16 _1g5..fdt > > 300684 Mar 10 03:16 _1g5.fdx > > 175 Mar 10 03:16 _1g5.fnm > > 125179984 Mar 10 03:28 _1g5.frq > > 89703062 Mar 10 03:28 _1g5.prx > > 2594360 Mar 10 03:28 _1g5.tii > > 184495760 Mar 10 03:28 _1g5.tis > > 64323505 Mar 10 04:09 _1gc.fdt > > 91020 Mar 10 04:09 _1gc.fdx > > 105283820 Mar 10 04:48 _1gf.fdt > > 148988 Mar 10 04:48 _1gf.fdx > > 175 Mar 10 04:09 _1gf.fnm > > 1491 Mar 10 04:09 _1gf.frq > > 4 Mar 10 04:09 _1gf.nrm > > 2388 Mar 10 04:09 _1gf.prx > > 254 Mar 10 04:09 _1gf.tii > > 15761 Mar 10 04:09 _1gf.tis > > 191035191 Mar 10 04:09 _1gg.fdt > > 270332 Mar 10 04:09 _1gg.fdx > > 175 Mar 10 04:09 _1gg.fnm > > 111958741 Mar 10 04:24 _1gg.frq > > 80645411 Mar 10 04:24 _1gg.prx > > 2349153 Mar 10 04:24 _1gg.tii > > 167494232 Mar 10 04:24 _1gg.tis > > 175 Mar 10 04:20 _1gh.fnm > > 10223275 Mar 10 04:20 _1gh.frq > > 4 Mar 10 04:20 _1gh.nrm > > 9056546 Mar 10 04:20 _1gh.prx > > 329012 Mar 10 04:20 _1gh.tii > > 23846511 Mar 10 04:20 _1gh.tis > > 175 Mar 10 04:28 _1gi.fnm > > 10221888 Mar 10 04:28 _1gi.frq > > 4 Mar 10 04:28 _1gi.nrm > > 9054280 Mar 10 04:28 _1gi.prx > > 328980 Mar 10 04:28 _1gi.tii > > 23843209 Mar 10 04:28 _1gi.tis > > 175 Mar 10 04:35 _1gj.fnm > > 10222776 Mar 10 04:35 _1gj.frq > > 4 Mar 10 04:35 _1gj.nrm > > 9054943 Mar 10 04:35 _1gj.prx > > 329060 Mar 10 04:35 _1gj.tii > > 23849395 Mar 10 04:35 _1gj.tis > > 175 Mar 10 04:42 _1gk.fnm > > 10220381 Mar 10 04:42 _1gk.frq > > 4 Mar 10 04:42 _1gk.nrm > > 9052810 Mar 10 04:42 _1gk.prx > > 329029 Mar 10 04:42 _1gk.tii > > 23845373 Mar 10 04:42 _1gk.tis > > 175 Mar 10 04:48 _1gl.fnm > > 9274170 Mar 10 04:48 _1gl.frq > > 4 Mar 10 04:48 _1gl.nrm > > 8226681 Mar 10 04:48 _1gl.prx > > 303327 Mar 10 04:48 _1gl.tii > > 21996826 Mar 10 04:48 _1gl.tis > > 22418126 Mar 10 04:58 _1gm.fdt > > 31732 Mar 10 04:58 _1gm.fdx > > 175 Mar 10 04:57 _1gm.fnm > > 10216672 Mar 10 04:57 _1gm.frq > > 4 Mar 10 04:57 _1gm.nrm > > 9049487 Mar 10 04:57 _1gm.prx > > 328813 Mar 10 04:57 _1gm.tii > > 23829627 Mar 10 04:57 _1gm.tis > > 175 Mar 10 04:58 _1gn.fnm > > 392014 Mar 10 04:58 _1gn.frq > > 4 Mar 10 04:58 _1gn.nrm > > 415225 Mar 10 04:58 _1gn.prx > > 24695 Mar 10 04:58 _1gn.tii > > 1816750 Mar 10 04:58 _1gn.tis > > 683 Mar 10 04:58 segments_7t > > 20 Mar 10 04:58 segments.gen > > 1935727800 Mar 9 11:17 _u1..fdt > > 2739180 Mar 9 11:17 _u1.fdx > > 175 Mar 9 11:15 _u1.fnm > > 1193583522 Mar 9 11:25 _u1.frq > > 817164507 Mar 9 11:25 _u1.prx > > 16547464 Mar 9 11:25 _u1..tii > > 1153764013 Mar 9 11:25 _u1.tis > > 1949493315 Mar 9 12:21 _x3.fdt > > 2758580 Mar 9 12:21 _x3.fdx > > 175 Mar 9 12:18 _x3.fnm > > 1202068425 Mar 9 12:29 _x3.frq > > 822963200 Mar 9 12:29 _x3.prx > > 16629485 Mar 9 12:29 _x3.tii > > 1159419149 Mar 9 12:29 _x3.tis > > > > > > Any ideas? I'm out of settings to tweak here. > > > > Cheers, > > Mark > > > > > > > > > > ----- Original Message ---- > > From: Michael McCandless <luc...@mikemccandless.com> > > To: java-user@lucene.apache.org > > Sent: Tuesday, 10 March, 2009 0:01:30 > > Subject: Re: A model for predicting indexing memory costs? > > > > > > mark harwood wrote: > > > >> > >> I've been building a large index (hundreds of millions) with mainly > structured data which consists of several fields with mostly unique values. > >> I've been hitting out of memory issues when doing periodic > commits/closes which I suspect is down to the sheer number of terms. > >> > >> I set the IndexWriter..setTermIndexInterval to 8 times the normal size > of 128 (an intervalof 1024) which delayed the onset of the issue but still > failed. > > > > I think that setting won't change how much RAM is used when writing. > > > >> I'd like to get a little more scientific about what to set here rather > than simply experimenting with settings and hoping it doesn't fail again. > >> > >> Does anyone have a decent model worked out for how much memory is > consumed at peak? I'm guessing the contributing factors are: > >> > >> * Numbers of fields > >> * Numbers of unique terms per field > >> * Numbers of segments? > > > > Number of net unique terms (across all fields) is a big driver, but also > net number of term occurrences, and how many docs. Lots of tiny docs take > more RAM than fewer large docs, when # occurrences are equal. > > > > But... how come setting IW's RAM buffer doesn't prevent the OOMs? IW > should simply flush when it's used that much RAM. > > > > I don't think number of segments is a factor. > > > > Though mergeFactor is, since during merging the SegmentMerger holds > SegmentReaders open, and int[] maps (if there are any deletes) for each > segment. Do you have a large merge taking place when you hit the OOMs? > > > > Mike > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
