Did you make any changes to your code between running on java 6 and 8?
The java file API has changed considerably since java 8.
If you -have- rewritten some of the file handling code in your indexing
process, make sure to explicitly close the streams you create, or use
the (since java 7) try-with-resources construct.
On 22/02/2017 16:18, Leonid Bolshinsky wrote:
I have a search engine based on Lucene 3.0.3 and I can't change the Lucene
version for reasons that are out of scope of this question. Now I have a
requirement to move from Java 6 to Java 8, however when I run the indexing
using Java 8 JVM, I hit "Too many open files issue" as below:
java.io.FileNotFoundException: /myIndex/_27c.fdx (Too many open files)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:85)
at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:192)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1559)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1578)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1155)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:832)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:807)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2155)
The ulimit on my system is 80.000 files and when I run lsof, I get that the
open files belong to the index and they are marked as (deleted) segments. I
am aware that there are tons of questions, answers and blogs about the
Lucene open files issue, however:
- I've already tried whatever is recommended (review the code looking
for open IndexWriters, increase ulimit, enable compound mode, decrease the
number of segments from 10 to 4, review the code for open files) without
success and
- The issue didn't appear with Java 6 so I am guessing that the problem
is in how Java 8 handles files.
Anybody faced this issue with Java 8 before? Any additional idea about how
to further troubleshoot this issue or what could be the cause?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
