Hi Michael,
I added a print statement to my code to output how many nodes have been
inserted. The script zips along until it gets up around 18.6K nodes. Then
the program halts when a Java exception is thrown, as shown below.
Inserting user 18629
Inserting user 18630
Inserting user 18631
Inserting user 18632
Inserting user 18633
Inserting user 18634
Inserting user 18635
Traceback (most recent call last):
File "jive_app_db_graph_construction.py", line 53, in <module>
node_idx['userid'][user_node['userid']] = user_node
File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/contextlib.py",
line 34, in __exit__
self.gen.throw(type, value, traceback)
File "<string>", line 11, in transaction
RuntimeError: An unknown error occured while handling a Java Exception
Strangely enough, the script returned a more explicit Out of Heap Memory
error once when I reran the script. It's bizarre that the failure output is
not consistent, given I start with a clean DB each time. Increasing the
amount of available heap memory seems like the reasonable place to start. If
you can let me know how to specify the amount of heap memory for the JVM
instantiated by the bindings, that'd be great.
I'm running on a MacBook Pro with 8 gigs of memory running OSX 10.6.8.
Here's the contents of the messages.log file.
2011-10-14 10:34:17.336-0700: Creating new db @ /Users/chris.diehl/Graphs/
JiveCommunityV5.0.ng/neostore
2011-10-14 10:34:17.419-0700: Opened [/Users/chris.diehl/Graphs/
JiveCommunityV5.0.ng/nioneo_logical.log.1] clean empty log, version=0,
lastTxId=1
2011-10-14 10:34:17.441-0700: Opened [/Users/chris.diehl/Graphs/
JiveCommunityV5.0.ng/index/lucene.log.1] clean empty log, version=0,
lastTxId=1
2011-10-14 10:34:17.443-0700: TM new log: tm_tx_log.1
2011-10-14 10:34:17.451-0700: --- CONFIGURATION START ---
2011-10-14 10:34:17.451-0700: Physical mem: 8192MB, Heap size: 123MB
2011-10-14 10:34:17.452-0700: Kernel version: Neo4j - Graph Database Kernel
1.5-SNAPSHOT (revision: 1.5.M02-4-g3d1e9d8)
2011-10-14 10:34:17.453-0700: Neo4j - Graph Database Kernel 1.5-SNAPSHOT
(revision: 1.5.M02-4-g3d1e9d8)
2011-10-14 10:34:17.456-0700: Operating System: Mac OS X; version: 10.6.8;
arch: x86_64; cpus: 8
2011-10-14 10:34:17.456-0700: VM Name: Java HotSpot(TM) 64-Bit Server VM
2011-10-14 10:34:17.456-0700: VM Vendor: Apple Inc.
2011-10-14 10:34:17.456-0700: VM Version: 20.1-b02-384
2011-10-14 10:34:17.456-0700: Boot Class Path:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsfd.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Frameworks/JavaRuntimeSupport.framework/Resources/Java/JavaRuntimeSupport.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/ui.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/laf.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/sunrsasign.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsse.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jce.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/charsets.jar
2011-10-14 10:34:17.457-0700: Class Path:
/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/antlr-2.7.7.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-analysis-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-commons-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-tree-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-util-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/blueprints-core-0.9.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/geronimo-jta_1.1_spec-1.1.1.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages
/neo4j/javalib/gremlin-1.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/groovy-1.8.1.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/jansi-1.5.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/jettison-1.3.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/lucene-core-3.1.0.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-cypher-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-graph-algo-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-graph-matching-1.5-SNAPSHOT.jar:/Users/chri
s.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-jmx-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-kernel-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-lucene-index-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-udc-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/pipes-0.7.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/scala-library-2.9.0-1.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/stax-api-1.0.1.jar
2011-10-14 10:34:17.457-0700: Library Path:
.:/Users/chris.diehl/Library/Java/Extensions:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
2011-10-14 10:34:17.459-0700: Garbage Collector: ParNew: [Par Eden Space,
Par Survivor Space]
2011-10-14 10:34:17.459-0700: Garbage Collector: ConcurrentMarkSweep: [Par
Eden Space, Par Survivor Space, CMS Old Gen, CMS Perm Gen]
2011-10-14 10:34:17.459-0700: VM Arguments: []
2011-10-14 10:34:17.459-0700:
2011-10-14 10:34:17.460-0700: create=true
2011-10-14 10:34:17.460-0700: logical_log=/Users/chris.diehl/Graphs/
JiveCommunityV5.0.ng/nioneo_logical.log
2011-10-14 10:34:17.460-0700: neo_store=/Users/chris.diehl/Graphs/
JiveCommunityV5.0.ng/neostore
2011-10-14 10:34:17.460-0700: neostore.nodestore.db.mapped_memory=20M
2011-10-14 10:34:17.460-0700:
neostore.propertystore.db.arrays.mapped_memory=130M
2011-10-14 10:34:17.460-0700:
neostore.propertystore.db.index.keys.mapped_memory=1M
2011-10-14 10:34:17.460-0700:
neostore.propertystore.db.index.mapped_memory=1M
2011-10-14 10:34:17.460-0700: neostore.propertystore.db.mapped_memory=90M
2011-10-14 10:34:17.460-0700:
neostore.propertystore.db.strings.mapped_memory=130M
2011-10-14 10:34:17.461-0700:
neostore.relationshipstore.db.mapped_memory=100M
2011-10-14 10:34:17.461-0700: node_auto_indexing=false
2011-10-14 10:34:17.461-0700: rebuild_idgenerators_fast=true
2011-10-14 10:34:17.461-0700: relationship_auto_indexing=false
2011-10-14 10:34:17.461-0700: store_dir=/Users/chris.diehl/Graphs/
JiveCommunityV5.0.ng
2011-10-14 10:34:17.461-0700: use_memory_mapped_buffers=true
2011-10-14 10:34:17.461-0700: --- CONFIGURATION END ---
2011-10-14 10:34:17.518-0700: Extension
org.neo4j.kernel.KernelExtension[kernel jmx] loaded ok
2011-10-14 10:34:17.520-0700: Extension
org.neo4j.kernel.KernelExtension[kernel udc] loaded ok
Chris
On Thu, Oct 13, 2011 at 8:37 PM, Michael Hunger <
michael.hun...@neotechnology.com> wrote:
> Hi Chris,
>
> What do you mean by "ground to a halt" ? Is it using all available CPU or
> heavy IO-waits (top, iostat, vmstat)?
>
> A thread dump (via jps + jstack ) would be great.
>
> You can probably pass in the config map into the Graph database call.
> Current config is logged to messages.log, please share that too as well as
> your system parameters (OS, CPU, RAM, Type of disk).
>
> Michael
>
> Am 13.10.2011 um 23:52 schrieb Chris Diehl:
>
> > Hey All,
> >
> > I just tested a Python script I've been working on to construct a
> > multi-relational graph in Neo4j. I am using the new Python bindings for
> > embedded Neo4j. After creating about 18K nodes, the database ground to a
> > halt.
> >
> > In the past, when using Gremlin, I would up the heap size for the JVM
> when
> > encountering problems. I'm not sure how to tweak settings for the JVM now
> > that I'm using the Python bindings to fire up the database. Can I do that
> > from the GraphDatabase() call? Or do I need to look elsewhere to tweak
> the
> > JVM parameters?
> >
> > If you have any other suggestions about parameters I should change, I'm
> all
> > ears.
> >
> > Thanks!
> >
> > Chris
> >
> > --
> > View this message in context:
> http://neo4j-community-discussions.438527.n3.nabble.com/Configuring-Neo4J-via-Python-bindings-tp3419961p3419961.html
> > Sent from the Neo4j Community Discussions mailing list archive at
> Nabble.com.
> > _______________________________________________
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
>
> _______________________________________________
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
_______________________________________________
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
