Hi Johan,
Thanks! Quick summary, it was definitely a config issue. Changing to
the recommend config settings makes the exceptions go away. Weird
though, those config values are for VM memory management. But the
exceptions looked like file I/O issues. How does Neo4J get around the
OS max file size limitation, does it just overflow into a new file?
I successfully ran the code resulting in roughly 190M nodes/properties
and 390M relationships injected in 35min (the condition in one of your
for loops probably has a typo, should have been personnumberOfPeople
Oh thanks, I thought that I was accessing a nodeID that didn't exist,
so I tried to go above the total number of nodes. I forgot to change
it back. Sorry.
I first tried on my laptop but ran out of disk space (with a similar
exception you got) since the full load requires 41GB of free space.
Could you please try the injection again (using similar configuration
and verifying you have enough disk space)?
I had enough disk space, 200GB+. I am still getting heap issues, I
don't have enough memory on my laptop, but I will run it on the cloud
later and they should go away.
Also (maybe not important since this is just a test but) relationships
can be treated as bi-directional/non directed using the enum constant
Direction.BOTH, so the two relationships a-KNOWS-b and b-KNOWS-a
can be represented with one a-KNOWS-b.
This is just for traversals, no? You can't define it as bi-directional
on creation, can you? I noticed that Neo doesn't explicitly define
cardinality (many-to-many, one-to-many, etc). Everything is supported
though. And the API is concise, which is prefereable.
Thanks again.
Regards,
Todd
Regards,
-Johan
On Wed, Sep 9, 2009 at 3:11 AM, Todd Stavishtoddstav...@gmail.com wrote:
He Johan,
What OS do you run this on? The FileChannel.write fail with an
exception saying buffer is too large is new to me.
OSX. Snow Leopard.
java -version
java version 1.6.0_15
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
I can try it on centos too. I was going to try to run in it in EC2,
but lost momentum and ideas to try.
Try decrease the amount of memory you give the store files (if you
have modified default configuration) and make sure you at least have
disk space to write out 1/10th of the dedicated memory for each store
file.
Plenty of disk space. Do you mean pre-allocate the database file? How
do you do that?
I will run some tests on this code first thing in the morning to see
if I can reproduce it.
Thanks. I will send you the input files. And other code.
Regards,
Todd
-Johan
On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavishtoddstav...@gmail.com wrote:
Hello All,
I hope that I haven't been too demanding of this list. I am pushing a
tight deadline. I can't seem to get Neo4J to scale above 1.5 million
nodes. The code is simple, just create a node, give it one property.
Then go back and randomly create relationships. I moved to the
BatchInserter too. Is there a way to use more than one DB? Is it a
file size limitation? Can I commit incrementally somehow? I looked at
the BatchInterter source, and there doesn't seem to be anything that I
can do.
Exceptions and demo code below.
Thanks,
Todd
Created 1516633 nodes in 33 seconds
Exception in thread main
org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write
record[1419120] @[188742960]
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111)
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123)
at
org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229)
at
org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503)
at
org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
at
org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
at
org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311)
at
com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78)
Caused by: java.io.IOException: Result too large
at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651)
at
