waaaay too many files in the index!

2009-02-03 Thread John Byrne
Hi, I've got a weird problem with a lucene index, using 2.3.1. The index contains 6660 files. I don't know how this happened.Maybe somone can tell me something about the files themselves? (examples below) On one day, between 10 and 40 of these files were being created every minute. The index

Re: waaaay too many files in the index!

2009-02-03 Thread Matthew Hall
Did you optimize your index? If not, depending on your merge factor, this could be a very normal index for you. -Matt John Byrne wrote: Hi, I've got a weird problem with a lucene index, using 2.3.1. The index contains 6660 files. I don't know how this happened.Maybe somone can tell me som

Re: waaaay too many files in the index!

2009-02-03 Thread Michael Stoppelman
On Tue, Feb 3, 2009 at 7:26 AM, John Byrne wrote: > Hi, > > I've got a weird problem with a lucene index, using 2.3.1. The index > contains 6660 files. I don't know how this happened.Maybe somone can tell me > something about the files themselves? (examples below) > > On one day, between 10 and 4

Re: waaaay too many files in the index!

2009-02-03 Thread Erick Erickson
What are your IndexWriter MergFactor and MergeDocs set to? Also, are the dates on all these files indicative of all being create during the same indexing run? Finally, how many documents are you indexing? Best Erick On Tue, Feb 3, 2009 at 10:26 AM, John Byrne wrote: > Hi, > > I've got a weird

Re: waaaay too many files in the index!

2009-02-04 Thread John Byrne
MergeFactor and MergeDocs are left at default values. The indexing is incremental, i.e. whenever someone adds or modifys a file to in svn repository, the lucene index is updated, and the writer/reader/searcher are refreshed (closed and opened again)., According to the svn logs for the time the

Re: waaaay too many files in the index!

2009-02-04 Thread Michael McCandless
These files are normal Lucene segment files (in compound file format). What's odd is that Lucene is not merging them down to a smaller set of segments. Have you done any advanced things, like customize the deletion or merge policy? When you close you writer, are you using just close()

Re: waaaay too many files in the index!

2009-02-04 Thread Michael McCandless
OK thanks for bringing closure. Mike John Byrne wrote: No I'm not messing with the delete or merge policy - but I think I know what went wrong though... We have 2 instances of the application, for failover. They are never supposed to be active at the same time, but I just discovered a

Re: waaaay too many files in the index!

2009-02-04 Thread John Byrne
No I'm not messing with the delete or merge policy - but I think I know what went wrong though... We have 2 instances of the application, for failover. They are never supposed to be active at the same time, but I just discovered a condition that can cause exactly that to happen. When we detec