Paul, I also would like to see automatic swapping/paging to disk as part of Neo4J, minimally when in "bulk insert" mode...and ideally in every usage scenario. I don't fully understand why the in-memory logs get so large and/or aren't backed by the on-disk log, or if they are, why they need to be kept in memory as well. Perhaps it isn't the transaction "stuff" that is taking up memory, but the graph itself?
Can any of the Neo team help provide some insight? Thanks! -----Original Message----- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Paul A. Jackson Sent: Thursday, July 08, 2010 1:35 PM To: (User@lists.neo4j.org) Subject: [Neo4j] OutOfMemory while populating large graph I have seen people discuss committing transactions after some microbatch of a few hundred records, but I thought this was optional. I thought Neo4J would automatically write out to disk as memory became full. Well, I encountered an OOM and want to make sure that I understand the reason. Was my understanding incorrect, or is there a parameter that I need to set to some limit, or is the problem them I am indexing as I go. The stack trace, FWIW, is: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.<init>(HashMap.java:209) at java.util.HashSet.<init>(HashSet.java:86) at org.neo4j.index.lucene.LuceneTransaction$TxCache.add(LuceneTransaction.java: 334) at org.neo4j.index.lucene.LuceneTransaction.insert(LuceneTransaction.java:93) at org.neo4j.index.lucene.LuceneTransaction.index(LuceneTransaction.java:59) at org.neo4j.index.lucene.LuceneXaConnection.index(LuceneXaConnection.java:94) at org.neo4j.index.lucene.LuceneIndexService.indexThisTx(LuceneIndexService.jav a:220) at org.neo4j.index.impl.GenericIndexService.index(GenericIndexService.java:54) at org.neo4j.index.lucene.LuceneIndexService.index(LuceneIndexService.java:209) at JiraLoader$JiraExtractor$Item.setNodeProperty(JiraLoader.java:321) at JiraLoader$JiraExtractor$Item.updateGraph(JiraLoader.java:240) Thanks, Paul Jackson _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
