We're having a heck of a time with too many file handles around here. When we create large indexes, we often get thousands of temporary files in a given index! Even worse, we just plain run out of file handles--even on boxes where we've upped the limits as much as we think we can! We've played around with various settings for the mergeFactor and maxMergeDocs, but these seem to have at best an indirect effect on the number of temporary files created.
I'm not very familiar with the Lucene file system yet, so can someone briefly explain how Lucene works on creating an index? How does it determine when to create a new temporary file in the index and when does it decide to compress the index? Also, is there any way we could limit the number of file handles used by Lucene? This is becoming a huge problem for us, so any insight would be appreciated. Thanks, Scott
