Hi, during stress tests, I got a "Too many open files" message (see detailed stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, Core i7 which handles the load with ease.
cat /proc/sys/fs/file-max 2399285 The parameter fs.file-max was already set very high, but I'll increase it further and see if it will happen again. fs.file-max = 6815744 fs.aio-max-nr = 1048576 Are there recommendations of linux kernel parameters? Or/and would you recommend other things to circumvent that, f.e. using multiple indexes instead of one? And it seems to me that finishing a Neo4j transaction will always trigger a LuceneTransaction.doCommit, even if there was no update on index (I try to avoid for the log nodes). Can this be switched off? Thanks and greetings Axel [1] Stack trace java.lang.RuntimeException: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Caused by: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86) at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) at org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) ... 12 more javax.transaction.xa.XAException: Unknown xid[GlobalId[NEOKERNL|5131995998687892543|61442], BranchId[ 52 49 52 49 52 49 ]] at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:470) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111) at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:533) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:648) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Mar 30, 2011 6:31:53 PM org.neo4j.kernel.impl.transaction.TxManager commit SEVERE: Unable to rollback transaction. Some resources may be commited others not. Neo4j kernel should be SHUTDOWN for resource maintance and transaction recovery ----> java.lang.RuntimeException: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Caused by: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86) at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) at org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) ... 12 more org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:104) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Caused by: javax.transaction.HeuristicMixedException: Unable to rollback ---> error code in commit: -1 ---> error code for rollback: 0 at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:662) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) ... 4 more _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user