Thanks Tom! Sounds like great fun working with such massive data sets :)
Mike http://blog.mikemccandless.com On Fri, May 20, 2011 at 7:03 PM, Burton-West, Tom <tburt...@umich.edu> wrote: > Hi Mike and Shai, > > > > I was able to index a few documents with the tieredMergePolicy but I was > hoping to build a large test index of about 700,000 documents to compare the > performance against our previous runs. I was hoping I would be able to > report on my results in time for the Lucene Revolution conference. > Unfortunately there was a power outage at our data center last week which > resulted in a node failure in one of our storage nodes and node rebalancing > for a cluster of 500 terabytes takes quite a while and totally messes up > performance measurements. (Our 6-8 terabytes of large scale search indexes > shares storage with the repository that holds the 480+ terabytes of page > images and metadata for the 8 million+ books). Hopefully I will be able to > run the tests when I get back. > > > > Tom > > > > From: Burton-West, Tom [mailto:tburt...@umich.edu] > Sent: Monday, May 09, 2011 4:10 PM > > To: dev@lucene.apache.org > Subject: RE: MergePolicy Thresholds > > > > Thanks again Shai and Mike. > > > > Am in the process of downloading and building r1099998. Should be able to > build a test index sometime this week. I’ll make some guesses on what > parameters to use based on our previous tests. > > > > Tom > > From: Shai Erera [mailto:ser...@gmail.com] > Sent: Saturday, May 07, 2011 11:33 PM > To: dev@lucene.apache.org > Subject: Re: MergePolicy Thresholds > > > > Hey Tom, > > Mike back-ported the changes to 3x, so you can try it out. > > FYI, > Shai > > On Tue, May 3, 2011 at 9:33 PM, Burton-West, Tom <tburt...@umich.edu> wrote: > > Thanks Shai and Mike! > > I'll keep an eye on LUCENE-1076. > > Tom > > -----Original Message----- > From: Michael McCandless [mailto:luc...@mikemccandless.com] > > Sent: Tuesday, May 03, 2011 11:15 AM > To: dev@lucene.apache.org > Subject: Re: MergePolicy Thresholds > > Thanks Shai! > > I'm way behind on my 3.x backports -- I'll try to do this soon. > > Mike > > http://blog.mikemccandless.com > > On Tue, May 3, 2011 at 8:10 AM, Shai Erera <ser...@gmail.com> wrote: >> I uploaded a patch to LUCENE-1076. >> >> Tom, apparently the patch I've attached before cannot be used, because >> there >> are dependencies (in earlier commits on LUCENE-1076) that need to be >> back-ported as well. So stay tuned on LUCENE-1076 for when it is safe to >> use >> this new MP. >> >> Shai >> >> On Tue, May 3, 2011 at 1:00 PM, Michael McCandless >> <luc...@mikemccandless.com> wrote: >>> >>> That'd be great, thanks :) >>> >>> Yes, let's iterate on the issue! But: it should still be open, I hope >>> (I didn't mean to close it yet, since it's not back ported)... >>> >>> Mike >>> >>> http://blog.mikemccandless.com >>> >>> On Tue, May 3, 2011 at 5:51 AM, Shai Erera <ser...@gmail.com> wrote: >>> > Mike, if you want, I can back-port it, as I've already started this >>> > when >>> > preparing the patch. >>> > >>> > I noticed that you added a "throws IOE" to IW.setInfoStream -- is it ok >>> > on >>> > 3x too? It'll be a backwards change. >>> > >>> > Maybe we should iterate on the issue? I can reopen. >>> > >>> > Shai >>> > >>> > On Tue, May 3, 2011 at 12:36 PM, Michael McCandless >>> > <luc...@mikemccandless.com> wrote: >>> >> >>> >> Looks good Shai! >>> >> >>> >> Comments below too: >>> >> >>> >> On Tue, May 3, 2011 at 5:29 AM, Shai Erera <ser...@gmail.com> wrote: >>> >> > Hi >>> >> > >>> >> > I looked into porting it to 3x, and prepared the attached patch. It >>> >> > only >>> >> > contains the new TieredMP and Test, as well as the necessary changes >>> >> > to >>> >> > LuceneTestCase and IndexWriter. I guess you can start with it (even >>> >> > just >>> >> > the >>> >> > MP and IW changes) to test it on your indexes. >>> >> > >>> >> > Mike, I saw that there were many more changes, as part of >>> >> > LUCENE-1076, >>> >> > done >>> >> > to the code. In particular, this MP is now the default (on trunk), >>> >> > so >>> >> > I >>> >> > guess many changes (to tests) were needed because of that. Do you >>> >> > remember, >>> >> > if apart from the changes I've included in the patch, other >>> >> > important >>> >> > changes w.r.t. this code? >>> >> >>> >> The only other changes I can think of were some verbosity improvements >>> >> to IndexWriter, to support the python script that can make a merge >>> >> movie from an infoStream output; but that can wait for when I >>> >> back-port to 3.x... >>> >> >>> >> > As we won't change the default MP on 3x, I'm guessing I don't need >>> >> > to >>> >> > port >>> >> > all the changes to 3x. >>> >> >>> >> Right, I think. >>> >> >>> >> Mike >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> >> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> >>> > >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
