Re: optimize with num segments > 1 index keeps growing

2011-09-13 Thread Michael McCandless
t; > 13.09.2011 12:44 > Please respond to > java-user@lucene.apache.org > > > > To > java-user@lucene.apache.org > cc > > Subject > Re: optimize with num segments > 1 index keeps growing > > > > > > > OK thanks for the infoStream output -

Re: optimize with num segments > 1 index keeps growing

2011-09-13 Thread v . sevel
OK. that worked. thanks, vincent Michael McCandless 13.09.2011 12:44 Please respond to java-user@lucene.apache.org To java-user@lucene.apache.org cc Subject Re: optimize with num segments > 1 index keeps growing OK thanks for the infoStream output -- it was very help

Re: optimize with num segments > 1 index keeps growing

2011-09-13 Thread Michael McCandless
(3.3):C19380892/4340449 > _m5eqg(3.3):C9760743 _m63c4(3.3):c1228 _m627z(3.3):c259956 > _m637u(3.3):c7630 _m63e2(3.3):c873 _m63ae(3.3):c932 _m63ah(3.3):c936 > _m63aj(3.3):c449 _m63ec(3.3):c570 _m63em(3.3):c886 _m63ce(3.3):c708 > _m63ds(3.3):c705 _m63c2(3.3):c1174 _m63di(3.3):c481 &g

Re: optimize with num segments > 1 index keeps growing

2011-09-12 Thread v . sevel
vincent Michael McCandless 12.09.2011 18:20 Please respond to java-user@lucene.apache.org To java-user@lucene.apache.org cc Subject Re: optimize with num segments > 1 index keeps growing Hmm... are you using IndexReader.numDeletedDocs to check? Did you commit from the

Re: optimize with num segments > 1 index keeps growing

2011-09-12 Thread Michael McCandless
> > > 09.09.2011 20:53 > Please respond to > java-user@lucene.apache.org > > > > To > java-user@lucene.apache.org > cc > > Subject > Re: optimize with num segments > 1 index keeps growing > > > > > > > TieredMergePolicy by default will on

Re: optimize with num segments > 1 index keeps growing

2011-09-12 Thread Michael McCandless
I committed a jdoc fix. Mike McCandless http://blog.mikemccandless.com On Mon, Sep 12, 2011 at 10:00 AM, Mark Miller wrote: > > On Sep 9, 2011, at 3:35 PM, Robert Muir wrote: > >> On Fri, Sep 9, 2011 at 3:07 PM, Uwe Schindler wrote: >>> Hi, >>> >>> This is still some kind of bug, because expun

Re: optimize with num segments > 1 index keeps growing

2011-09-12 Thread Mark Miller
On Sep 9, 2011, at 3:35 PM, Robert Muir wrote: > On Fri, Sep 9, 2011 at 3:07 PM, Uwe Schindler wrote: >> Hi, >> >> This is still some kind of bug, because expungeDeletes is documented to >> remove all deletes. Maybe we need to modify MergePolicy? >> > > we should correct the javadocs for exp

Re: optimize with num segments > 1 index keeps growing

2011-09-10 Thread v . sevel
To java-user@lucene.apache.org cc Subject Re: optimize with num segments > 1 index keeps growing TieredMergePolicy by default will only merge a segment if it has > 10% deletions. Can you try calling .setExpungeDeletesPctAllowed(0.0) and then expunge again? Mike McCandless

Re: optimize with num segments > 1 index keeps growing

2011-09-09 Thread Robert Muir
On Fri, Sep 9, 2011 at 3:07 PM, Uwe Schindler wrote: > Hi, > > This is still some kind of bug, because expungeDeletes is documented to > remove all deletes. Maybe we need to modify MergePolicy? > we should correct the javadocs for expungeDeletes here I think: so that its more consistent with the

Re: optimize with num segments > 1 index keeps growing

2011-09-09 Thread Uwe Schindler
5'347 _jyb4c.fnm > 09.09.2011 19:03 6'474'023'590 _jyb4c.frq > 09.09.2011 19:03 3'699'406'141 _jyb4c.prx > 09.09.2011 19:03 37'900'657 _jyb4c.tii > 09.09.2011 19:03 3'380'266'875 _jyb4c.tis > 09.09.

Re: optimize with num segments > 1 index keeps growing

2011-09-09 Thread Michael McCandless
'812 _jyb4c.fdx > 09.09.2011  18:27             5'347 _jyb4c.fnm > 09.09.2011  19:03     6'474'023'590 _jyb4c.frq > 09.09.2011  19:03     3'699'406'141 _jyb4c.prx > 09.09.2011  19:03        37'900'657 _jyb4c.tii > 09.09.2

RE: optimize with num segments > 1 index keeps growing

2011-09-09 Thread v . sevel
b5t.cfs 09.09.2011 19:31 9'545 _jyb5u.cfs 09.09.2011 19:31 8'963 _jyb5v.cfs 09.09.2011 19:31 9'250 _jyb5w.cfs 09.09.2011 19:31 9'047 _jyb5x.cfs 09.09.2011 19:3111'253 _jyb5y.cfs 09.09.2011 19:3111

RE: optimize with num segments > 1 index keeps growing

2011-07-21 Thread Uwe Schindler
o: java-user@lucene.apache.org > Subject: Re: optimize with num segments > 1 index keeps growing > > Hi, thanks for this explanation. > so what is the best solution: merge the large segment (how can I do that) or > work with many segments (10?) so that I will avoid have this &qu

Re: optimize with num segments > 1 index keeps growing

2011-07-21 Thread v . sevel
raterie - 1204 Genève - Suisse T +41 22 709 3376 - F +41 22 709 3782 www.lombardodier.com Simon Willnauer 21.07.2011 20:06 Please respond to java-user@lucene.apache.org To java-user@lucene.apache.org cc Subject Re: optimize with num segments > 1 index keeps growing so the pro

Re: optimize with num segments > 1 index keeps growing

2011-07-21 Thread Simon Willnauer
ene.apache.org > > > > To > java-user@lucene.apache.org > cc > > Subject > Re: optimize with num segments > 1 index keeps growing > > > > > > > A write.lock file with timestamp of 13:58 is in all the listings. The > first thing I'd try is to add s

Re: optimize with num segments > 1 index keeps growing

2011-07-21 Thread v . sevel
.2011 13:58 0 write.lock > 21.07.2011 16:0632'901'423'750 _7e3hz.fdt > 21.07.2011 16:06 149'582'052 _7e3hz.fdx > 21.07.2011 15:42 5'002 _7e3hz.fnm > 21.07.2011 16:46 8'608'541'177 _7e3hz.frq > 21.07.2

Re: optimize with num segments > 1 index keeps growing

2011-07-21 Thread Ian Lea
      0 write.lock > 21.07.2011  16:06    32'901'423'750 _7e3hz.fdt > 21.07.2011  16:06       149'582'052 _7e3hz.fdx > 21.07.2011  15:42             5'002 _7e3hz.fnm > 21.07.2011  16:46     8'608'541'177 _7e3hz.frq > 21.07.2011  16:46    

Re: optimize with num segments > 1 index keeps growing

2011-07-21 Thread v . sevel
7;608'541'177 _7e3hz.frq 21.07.2011 16:46 4'392'616'115 _7e3hz.prx 21.07.2011 16:4650'571'856 _7e3hz.tii 21.07.2011 16:46 4'515'914'658 _7e3hz.tis 10 File(s) 50'618'654'908 bytes this means that with

Re: optimize with num segments > 1 index keeps growing

2011-07-20 Thread Simon Willnauer
On Wed, Jul 20, 2011 at 2:00 PM, wrote: > Hi, > > I index several millions small documents per day. each day, I remove some > of the older documents to keep the index at a stable number of documents. > after each purge, I commit then I optimize the index. what I found is that > if I keep optimizi