This is looking like its just a Lucene oddity you get when adding a single doc due to some changes with the NRT stuff.
Mark Miller wrote: > Okay - I'm sorry - serves me right for working sick. > > Now that I have put on my glasses and correctly tagged my two eclipse tests: > > It still appears that trunk likes to use more RAM. > > I switched both tests to one million iterations and watched the heap. > > The test from the build around may 5th (I promise :) ) regularly GC's > down to about 70-80MB after a fair time > of running. It doesn't appear to climb - keeps GC'ing back to 70-80 > (after starting at by GC'ing down to 40 for a bit). > > The test from trunk, after a fair time of running, keeps GC'ing down to > about 120-150MB - 150 at the end, slowly working its > way up from 90-110 at the beginning. > > Don't know what that means yet - but it appears trunk likes to use a bit > more RAM while indexing. Odd that its so much more because these docs > are tiny: > > String[] fields = {"text","simple" > ,"text","test" > ,"text","how now brown cow" > ,"text","what's that?" > ,"text","radical!" > ,"text","what's all this about, anyway?" > ,"text","just how fast is this text indexing?" > }; > > Mark Miller wrote: > >> Okay, I juggled the tests in eclipse and flipped the results. So they >> make sense. >> >> Sorry - goose chase on this one. >> >> Yonik Seeley wrote: >> >> >>> I don't see this with trunk... I just tried TestIndexingPerformance >>> with 1M docs, and it seemed to work fine. >>> Memory use stabilized at 40MB. >>> Most memory use was for indexing (not analysis). >>> char[] topped out at 4.5MB >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> >>> On Tue, Oct 6, 2009 at 12:31 PM, Mark Miller <markrmil...@gmail.com> wrote: >>> >>> >>> >>>> Yeah - I was wondering about that ... not sure how these guys are >>>> stacking up ... >>>> >>>> Yonik Seeley wrote: >>>> >>>> >>>> >>>>> TestIndexingPerformance? >>>>> What the heck... that's not even multi-threaded! >>>>> >>>>> -Yonik >>>>> http://www.lucidimagination.com >>>>> >>>>> >>>>> >>>>> On Tue, Oct 6, 2009 at 12:17 PM, Mark Miller <markrmil...@gmail.com> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Darnit - didn't finish that email. This is after running your old short >>>>>> doc perf test for 10,000 iterations. You see the same thing with 1000 >>>>>> iterations but much less pronounced eg gettin' worse with more >>>>>> iterations. >>>>>> >>>>>> Mark Miller wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> A little before and after. The before is around may 5th'is - the after >>>>>>> is trunk. >>>>>>> >>>>>>> http://myhardshadow.com/memanalysis/before.png >>>>>>> http://myhardshadow.com/memanalysis/after.png >>>>>>> >>>>>>> Mark Miller wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Took a peak at the checkout around the time he says he's using. >>>>>>>> >>>>>>>> CharTokenizer appears to be holding onto much large char[] arrays now >>>>>>>> than before. Same with snowball.Among - used to be almost nothing, now >>>>>>>> its largio. >>>>>>>> >>>>>>>> The new TokenStream stuff appears to be clinging. Needs to find some >>>>>>>> inner peace. >>>>>>>> >>>>>>>> >>>>>>>> >> >> > > > -- - Mark http://www.lucidimagination.com