Re: Excessive reads while doing commit in lucene

2024-09-04 Thread Michael McCandless
It's odd to have a ~500X difference in writes versus reads. Are you sure? Is it possible you are also opening IndexReaders and searching the commit points? Lucene does re-read previously written (already indexed) documents during segment merges. But at default settings (as long as you did not ch

Re: Excessive reads while doing commit in lucene

2024-09-04 Thread Robert Muir
On Wed, Sep 4, 2024 at 7:07 AM Gopal Sharma wrote: > > Hi Team, > > I am using aws efs to store a lucene index. That's the issue, don't use NFS! - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional co

Excessive reads while doing commit in lucene

2024-09-04 Thread Gopal Sharma
Hi Team, I am using aws efs to store a lucene index. While indexing for around 200 millions records. The efs read iops went till 20,097 GB's and the cost for efs went too high. Whereas the write iops was only 56GB In my use case i am committing every 100k records (because in my test scenarios com