Paul, I also would like to see automatic swapping/paging to disk as part of
Neo4J, minimally when in "bulk insert" mode...and ideally in every usage
scenario.  I don't fully understand why the in-memory logs get so large
and/or aren't backed by the on-disk log, or if they are, why they need to be
kept in memory as well.  Perhaps it isn't the transaction "stuff" that is
taking up memory, but the graph itself?

Can any of the Neo team help provide some insight?

Thanks!


-----Original Message-----
From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On
Behalf Of Paul A. Jackson
Sent: Thursday, July 08, 2010 1:35 PM
To: (User@lists.neo4j.org)
Subject: [Neo4j] OutOfMemory while populating large graph

I have seen people discuss committing transactions after some microbatch of
a few hundred records, but I thought this was optional.  I thought Neo4J
would automatically write out to disk as memory became full.  Well, I
encountered an OOM and want to make sure that I understand the reason.  Was
my understanding incorrect, or is there a parameter that I need to set to
some limit, or is the problem them I am indexing as I go.  The stack trace,
FWIW, is:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
            at java.util.HashMap.<init>(HashMap.java:209)
            at java.util.HashSet.<init>(HashSet.java:86)
            at
org.neo4j.index.lucene.LuceneTransaction$TxCache.add(LuceneTransaction.java:
334)
            at
org.neo4j.index.lucene.LuceneTransaction.insert(LuceneTransaction.java:93)
            at
org.neo4j.index.lucene.LuceneTransaction.index(LuceneTransaction.java:59)
            at
org.neo4j.index.lucene.LuceneXaConnection.index(LuceneXaConnection.java:94)
            at
org.neo4j.index.lucene.LuceneIndexService.indexThisTx(LuceneIndexService.jav
a:220)
            at
org.neo4j.index.impl.GenericIndexService.index(GenericIndexService.java:54)
            at
org.neo4j.index.lucene.LuceneIndexService.index(LuceneIndexService.java:209)
            at
JiraLoader$JiraExtractor$Item.setNodeProperty(JiraLoader.java:321)
            at
JiraLoader$JiraExtractor$Item.updateGraph(JiraLoader.java:240)

Thanks,
Paul Jackson
_______________________________________________
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

_______________________________________________
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Reply via email to