Hi Mike and Shai, I was able to index a few documents with the tieredMergePolicy but I was hoping to build a large test index of about 700,000 documents to compare the performance against our previous runs. I was hoping I would be able to report on my results in time for the Lucene Revolution conference. Unfortunately there was a power outage at our data center last week which resulted in a node failure in one of our storage nodes and node rebalancing for a cluster of 500 terabytes takes quite a while and totally messes up performance measurements. (Our 6-8 terabytes of large scale search indexes shares storage with the repository that holds the 480+ terabytes of page images and metadata for the 8 million+ books). Hopefully I will be able to run the tests when I get back.
Tom From: Burton-West, Tom [mailto:tburt...@umich.edu] Sent: Monday, May 09, 2011 4:10 PM To: dev@lucene.apache.org Subject: RE: MergePolicy Thresholds Thanks again Shai and Mike. Am in the process of downloading and building r1099998. Should be able to build a test index sometime this week. I'll make some guesses on what parameters to use based on our previous tests. Tom From: Shai Erera [mailto:ser...@gmail.com] Sent: Saturday, May 07, 2011 11:33 PM To: dev@lucene.apache.org Subject: Re: MergePolicy Thresholds Hey Tom, Mike back-ported the changes to 3x, so you can try it out. FYI, Shai On Tue, May 3, 2011 at 9:33 PM, Burton-West, Tom <tburt...@umich.edu<mailto:tburt...@umich.edu>> wrote: Thanks Shai and Mike! I'll keep an eye on LUCENE-1076. Tom -----Original Message----- From: Michael McCandless [mailto:luc...@mikemccandless.com<mailto:luc...@mikemccandless.com>] Sent: Tuesday, May 03, 2011 11:15 AM To: dev@lucene.apache.org<mailto:dev@lucene.apache.org> Subject: Re: MergePolicy Thresholds Thanks Shai! I'm way behind on my 3.x backports -- I'll try to do this soon. Mike http://blog.mikemccandless.com On Tue, May 3, 2011 at 8:10 AM, Shai Erera <ser...@gmail.com<mailto:ser...@gmail.com>> wrote: > I uploaded a patch to LUCENE-1076. > > Tom, apparently the patch I've attached before cannot be used, because there > are dependencies (in earlier commits on LUCENE-1076) that need to be > back-ported as well. So stay tuned on LUCENE-1076 for when it is safe to use > this new MP. > > Shai > > On Tue, May 3, 2011 at 1:00 PM, Michael McCandless > <luc...@mikemccandless.com<mailto:luc...@mikemccandless.com>> wrote: >> >> That'd be great, thanks :) >> >> Yes, let's iterate on the issue! But: it should still be open, I hope >> (I didn't mean to close it yet, since it's not back ported)... >> >> Mike >> >> http://blog.mikemccandless.com >> >> On Tue, May 3, 2011 at 5:51 AM, Shai Erera >> <ser...@gmail.com<mailto:ser...@gmail.com>> wrote: >> > Mike, if you want, I can back-port it, as I've already started this when >> > preparing the patch. >> > >> > I noticed that you added a "throws IOE" to IW.setInfoStream -- is it ok >> > on >> > 3x too? It'll be a backwards change. >> > >> > Maybe we should iterate on the issue? I can reopen. >> > >> > Shai >> > >> > On Tue, May 3, 2011 at 12:36 PM, Michael McCandless >> > <luc...@mikemccandless.com<mailto:luc...@mikemccandless.com>> wrote: >> >> >> >> Looks good Shai! >> >> >> >> Comments below too: >> >> >> >> On Tue, May 3, 2011 at 5:29 AM, Shai Erera >> >> <ser...@gmail.com<mailto:ser...@gmail.com>> wrote: >> >> > Hi >> >> > >> >> > I looked into porting it to 3x, and prepared the attached patch. It >> >> > only >> >> > contains the new TieredMP and Test, as well as the necessary changes >> >> > to >> >> > LuceneTestCase and IndexWriter. I guess you can start with it (even >> >> > just >> >> > the >> >> > MP and IW changes) to test it on your indexes. >> >> > >> >> > Mike, I saw that there were many more changes, as part of >> >> > LUCENE-1076, >> >> > done >> >> > to the code. In particular, this MP is now the default (on trunk), so >> >> > I >> >> > guess many changes (to tests) were needed because of that. Do you >> >> > remember, >> >> > if apart from the changes I've included in the patch, other important >> >> > changes w.r.t. this code? >> >> >> >> The only other changes I can think of were some verbosity improvements >> >> to IndexWriter, to support the python script that can make a merge >> >> movie from an infoStream output; but that can wait for when I >> >> back-port to 3.x... >> >> >> >> > As we won't change the default MP on 3x, I'm guessing I don't need to >> >> > port >> >> > all the changes to 3x. >> >> >> >> Right, I think. >> >> >> >> Mike >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: >> >> dev-unsubscr...@lucene.apache.org<mailto:dev-unsubscr...@lucene.apache.org> >> >> For additional commands, e-mail: >> >> dev-h...@lucene.apache.org<mailto:dev-h...@lucene.apache.org> >> >> >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: >> dev-unsubscr...@lucene.apache.org<mailto:dev-unsubscr...@lucene.apache.org> >> For additional commands, e-mail: >> dev-h...@lucene.apache.org<mailto:dev-h...@lucene.apache.org> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org<mailto:dev-unsubscr...@lucene.apache.org> For additional commands, e-mail: dev-h...@lucene.apache.org<mailto:dev-h...@lucene.apache.org> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org<mailto:dev-unsubscr...@lucene.apache.org> For additional commands, e-mail: dev-h...@lucene.apache.org<mailto:dev-h...@lucene.apache.org>
