I have a search engine based on Lucene 3.0.3 and I can't change the Lucene
version for reasons that are out of scope of this question. Now I have a
requirement to move from Java 6 to Java 8, however when I run the indexing
using Java 8 JVM, I hit "Too many open files issue" as below:
java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open files)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:85)
at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:192)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1559)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1578)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1155)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:832)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:807)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2155)
The ulimit on my system is 80.000 files and when I run lsof, I get that the
open files belong to the index and they are marked as (deleted) segments. I
am aware that there are tons of questions, answers and blogs about the
Lucene open files issue, however:
- I've already tried whatever is recommended (review the code looking
for open IndexWriters, increase ulimit, enable compound mode, decrease the
number of segments from 10 to 4, review the code for open files) without
success and
- The issue didn't appear with Java 6 so I am guessing that the problem
is in how Java 8 handles files.
Anybody faced this issue with Java 8 before? Any additional idea about how
to further troubleshoot this issue or what could be the cause?
