commit frequency guideline?

2016-11-30 Thread Rob Audenaerde
Hi all, Currently we call commit() many times on our index (about 5M docs, where some 10.000-100.000 modifications during the day). The commit times typically get more expensive when the index grows, up to several seconds, so we want to reduce the number of calls. (Historically, we had Lucene com

Re: commit frequency guideline?

2016-11-30 Thread Michael McCandless
What do you mean by "Lucene complain about too-many uncommitted docs"? Lucene does not really care how frequently you commit... How frequently you commit is really your choice, i.e. what risk you see of power loss / OS crash vs the cost (not just in CPU/IO work for the computer, but in the users

Re: commit frequency guideline?

2016-11-30 Thread Rob Audenaerde
Thanks for the quick reply! >What do you mean by "Lucene complain about too-many uncommitted docs"? --> good question, I was thoughtlessly echoing words from my colleague. I asked him and he said that it was about taking very long to commit and memory issues. So maybe this wasn't the best opening

Re: commit frequency guideline?

2016-12-02 Thread Michael McCandless
On Wed, Nov 30, 2016 at 9:37 AM, Rob Audenaerde wrote: > Thanks for the quick reply! > >>What do you mean by "Lucene complain about too-many uncommitted docs"? > > --> good question, I was thoughtlessly echoing words from my colleague. I > asked him and he said that it was about taking very long t