>>> See more information at >>> http://components.neo4j.org/neo4j-index/apidocs/index.html >>> >>> Are you using the "index" component btw (artifactId: neo4j-index)? >>> >>> >> Yes. >> >> Is it possible that the problem is caused due to the fact that we >> perform several writes in both the graph and the index and we don't >> commit and that we read in between the transaction commits? >> > No that shouldn't cause any problems. This looks like a bug then. > Could you give us more information about your code, some snippets > perhaps or a test case which can trigger it again? > In fact I've tried increasing the ulimit -n parameter of the linux machine where I was running the experiment (it was 1024 and I set it to 10000). It appears to work so far (still running the test). What I understand is that there are many open files from lucene due to the fact that no optimize() is called on time. Since optimize() is not exposed from the LuceneIndexService, should I expect that it is handled internally?
The project I'm running is rather large, but at some point I will try to prepare a script in order to replicate the error. >> >>> 2010/1/26 Symeon (Akis) Papadopoulos <papa...@iti.gr>: >>> >>> >>>> Hi all >>>> >>>> While populating a Neo graph, I got the attached exception >>>> (org.apache.lucene.index.MergePolicy$MergeException. See file for details: >>>> I >>>> replaced a local path with [some-local-path]). >>>> My setup is like this: I want to benchmark Neo4j for some operations, >>>> amongst which is graph loading. >>>> So, I try to load graphs of various sizes to Neo4j. Up to size 1M edges the >>>> graphs are loaded without any problem, but then this exception is thrown. I >>>> suspect this has to do with the transaction management in my program, which >>>> is handled by a BatchTxManager (a class written by me, thus highly likely >>>> to >>>> be the source of trouble). Does the exception ring any bells? What could I >>>> try out in order to identify the problem? >>>> >>>> Best regards, >>>> Akis >>>> >>>> >>>> >>>> Exception in thread "Lucene Merge Thread #0" >>>> org.apache.lucene.index.MergePolicy$MergeException: >>>> java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many open >>>> files) >>>> at >>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351) >>>> at >>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315) >>>> Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too >>>> many open files) >>>> at java.io.RandomAccessFile.open(Native Method) >>>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >>>> at >>>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78) >>>> at >>>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108) >>>> at >>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:94) >>>> at >>>> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70) >>>> at >>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) >>>> at >>>> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:111) >>>> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638) >>>> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608) >>>> at >>>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686) >>>> at >>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979) >>>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614) >>>> at >>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235) >>>> at >>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291) >>>> Exception in thread "main" >>>> org.neo4j.kernel.impl.transaction.TransactionFailureException: Unable to >>>> commit transaction >>>> at >>>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:331) >>>> at >>>> org.neo4j.util.SimpleBatchTxManager.beginReadOperation(SimpleBatchTxManager.java:94) >>>> at >>>> graph.implementation.neo.STSGraphNeoImpl.getUserNeo(STSGraphNeoImpl.java:463) >>>> at >>>> graph.implementation.neo.STSGraphNeoImpl.increaseUserTagFreq(STSGraphNeoImpl.java:824) >>>> at >>>> graph.benchmark.LoadGraphBenchmark.runRealBenchmark(LoadGraphBenchmark.java:433) >>>> at >>>> graph.benchmark.LoadGraphBenchmark.main(LoadGraphBenchmark.java:489) >>>> Caused by: java.lang.RuntimeException: Unable to close lucene writer >>>> org.apache.lucene.index.indexwri...@32a4eb93 >>>> at >>>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:413) >>>> at >>>> org.neo4j.index.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:197) >>>> at >>>> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316) >>>> at >>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399) >>>> at >>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >>>> at >>>> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514) >>>> at >>>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:573) >>>> at >>>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543) >>>> at >>>> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102) >>>> at >>>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:316) >>>> ... 5 more >>>> Caused by: java.io.IOException: directory '/[some-local-path]/resource_id' >>>> exists and is a directory, but cannot be listed: list() returned null >>>> at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516) >>>> at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531) >>>> at >>>> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307) >>>> at >>>> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300) >>>> at >>>> org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192) >>>> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183) >>>> at >>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2190) >>>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153) >>>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117) >>>> at >>>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:409) >>>> ... 14 more >>>> >>>> >>>> _______________________________________________ >>>> Neo mailing list >>>> User@lists.neo4j.org >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>>> >>>> >>>> >>> >>> >>> >> _______________________________________________ >> Neo mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> >> > > > > _______________________________________________ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
