Lucene cpu utilization & scoring

2021-08-20 Thread Varun Sharma
Hi, We have a large index that we divide into X lucene indices - we use lucene 6.5.0. On each of our serving machines serves 8 lucene indices in parallel. We are getting realtime updates to each of these 8 indices. We are seeing a couple of things: a) When we turn off realtime updates, performanc

Re: Lucene cpu utilization & scoring

2021-08-20 Thread Michael Sokolov
I think the usual usage pattern is to *refresh* frequently and commit less frequently. Is there a reason you need to commit often? You may also have overlooked this newish method: MergePolicy.findFullFlushMerges If you implement that, you can tell IndexWriter to (for example) merge multiple small

Re: Lucene cpu utilization & scoring

2021-08-20 Thread Varun Sharma
Thanks, Michael. Its good to know that scorers are also doing matching. I will check and verify whether the scores returned are 0 or not. Just to give some background, we have two setups: a) Old setup - Each machine serves a single lucene index which has roughly 30'ish segments with realtime updat