Re: [Neo] Neo4J Scalability
Hi Johan, Thanks! Quick summary, it was definitely a config issue. Changing to the recommend config settings makes the exceptions go away. Weird though, those config values are for VM memory management. But the exceptions looked like file I/O issues. How does Neo4J get around the OS max file size limitation, does it just overflow into a new file? I successfully ran the code resulting in roughly 190M nodes/properties and 390M relationships injected in 35min (the condition in one of your for loops probably has a typo, should have been personnumberOfPeople Oh thanks, I thought that I was accessing a nodeID that didn't exist, so I tried to go above the total number of nodes. I forgot to change it back. Sorry. I first tried on my laptop but ran out of disk space (with a similar exception you got) since the full load requires 41GB of free space. Could you please try the injection again (using similar configuration and verifying you have enough disk space)? I had enough disk space, 200GB+. I am still getting heap issues, I don't have enough memory on my laptop, but I will run it on the cloud later and they should go away. Also (maybe not important since this is just a test but) relationships can be treated as bi-directional/non directed using the enum constant Direction.BOTH, so the two relationships a-KNOWS-b and b-KNOWS-a can be represented with one a-KNOWS-b. This is just for traversals, no? You can't define it as bi-directional on creation, can you? I noticed that Neo doesn't explicitly define cardinality (many-to-many, one-to-many, etc). Everything is supported though. And the API is concise, which is prefereable. Thanks again. Regards, Todd Regards, -Johan On Wed, Sep 9, 2009 at 3:11 AM, Todd Stavishtoddstav...@gmail.com wrote: He Johan, What OS do you run this on? The FileChannel.write fail with an exception saying buffer is too large is new to me. OSX. Snow Leopard. java -version java version 1.6.0_15 Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219) Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode) I can try it on centos too. I was going to try to run in it in EC2, but lost momentum and ideas to try. Try decrease the amount of memory you give the store files (if you have modified default configuration) and make sure you at least have disk space to write out 1/10th of the dedicated memory for each store file. Plenty of disk space. Do you mean pre-allocate the database file? How do you do that? I will run some tests on this code first thing in the morning to see if I can reproduce it. Thanks. I will send you the input files. And other code. Regards, Todd -Johan On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavishtoddstav...@gmail.com wrote: Hello All, I hope that I haven't been too demanding of this list. I am pushing a tight deadline. I can't seem to get Neo4J to scale above 1.5 million nodes. The code is simple, just create a node, give it one property. Then go back and randomly create relationships. I moved to the BatchInserter too. Is there a way to use more than one DB? Is it a file size limitation? Can I commit incrementally somehow? I looked at the BatchInterter source, and there doesn't seem to be anything that I can do. Exceptions and demo code below. Thanks, Todd Created 1516633 nodes in 33 seconds Exception in thread main org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write record[1419120] @[188742960] at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503) at org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311) at com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78) Caused by: java.io.IOException: Result too large at sun.nio.ch.FileDispatcher.pwrite0(Native Method) at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651) at
Re: [Neo] Neo4J Scalability
On Thu, Sep 10, 2009 at 2:40 PM, Todd Stavish toddstav...@gmail.com wrote: Thanks! Quick summary, it was definitely a config issue. Changing to the recommend config settings makes the exceptions go away. Weird though, those config values are for VM memory management. But the exceptions looked like file I/O issues. How does Neo4J get around the OS max file size limitation, does it just overflow into a new file? Yes, strange with that IOException instead of an OutOfMemory. Neo4j does not overflow into a new file at the moment (this may be implemented in Neo4j 2.0) but this has not been a problem since if you have a large graph/data set they are typically deployed on a server that can handle large files. I had enough disk space, 200GB+. I am still getting heap issues, I don't have enough memory on my laptop, but I will run it on the cloud later and they should go away. May still be possible to get it to run on less memory but the time to load the data will increase. For a 2G heap try: neostore.nodestore.db.mapped_memory=800M neostore.relationshipstore.db.mapped_memory=800M neostore.propertystore.db.mapped_memory=100M neostore.propertystore.db.index.mapped_memory=1M neostore.propertystore.db.index.keys.mapped_memory=1M neostore.propertystore.db.strings.mapped_memory=150M neostore.propertystore.db.arrays.mapped_memory=1M Regards, -Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Neo4J Scalability
Todd, I successfully ran the code resulting in roughly 190M nodes/properties and 390M relationships injected in 35min (the condition in one of your for loops probably has a typo, should have been personnumberOfPeople and then more relationships would have been created). This was on a Linux machine with 8G RAM running with a 5G heap and the following configuration: neostore.nodestore.db.mapped_memory=1700M neostore.relationshipstore.db.mapped_memory=2000M neostore.propertystore.db.mapped_memory=100M neostore.propertystore.db.index.mapped_memory=1M neostore.propertystore.db.index.keys.mapped_memory=1M neostore.propertystore.db.strings.mapped_memory=250M neostore.propertystore.db.arrays.mapped_memory=130M I first tried on my laptop but ran out of disk space (with a similar exception you got) since the full load requires 41GB of free space. Could you please try the injection again (using similar configuration and verifying you have enough disk space)? Also (maybe not important since this is just a test but) relationships can be treated as bi-directional/non directed using the enum constant Direction.BOTH, so the two relationships a-KNOWS-b and b-KNOWS-a can be represented with one a-KNOWS-b. Regards, -Johan On Wed, Sep 9, 2009 at 3:11 AM, Todd Stavishtoddstav...@gmail.com wrote: He Johan, What OS do you run this on? The FileChannel.write fail with an exception saying buffer is too large is new to me. OSX. Snow Leopard. java -version java version 1.6.0_15 Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219) Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode) I can try it on centos too. I was going to try to run in it in EC2, but lost momentum and ideas to try. Try decrease the amount of memory you give the store files (if you have modified default configuration) and make sure you at least have disk space to write out 1/10th of the dedicated memory for each store file. Plenty of disk space. Do you mean pre-allocate the database file? How do you do that? I will run some tests on this code first thing in the morning to see if I can reproduce it. Thanks. I will send you the input files. And other code. Regards, Todd -Johan On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavishtoddstav...@gmail.com wrote: Hello All, I hope that I haven't been too demanding of this list. I am pushing a tight deadline. I can't seem to get Neo4J to scale above 1.5 million nodes. The code is simple, just create a node, give it one property. Then go back and randomly create relationships. I moved to the BatchInserter too. Is there a way to use more than one DB? Is it a file size limitation? Can I commit incrementally somehow? I looked at the BatchInterter source, and there doesn't seem to be anything that I can do. Exceptions and demo code below. Thanks, Todd Created 1516633 nodes in 33 seconds Exception in thread main org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write record[1419120] @[188742960] at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503) at org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311) at com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78) Caused by: java.io.IOException: Result too large at sun.nio.ch.FileDispatcher.pwrite0(Native Method) at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105) public class GraphIngestBatch { public static void main(String[] args) { try { // Create graph database service BatchInserter neo = new BatchInserterImpl(graphDB/, BatchInserterImpl.loadProperties(neo.props)); // Read raw text for name data FileReader namesFile = new
[Neo] Neo4J Scalability
Hello All, I hope that I haven't been too demanding of this list. I am pushing a tight deadline. I can't seem to get Neo4J to scale above 1.5 million nodes. The code is simple, just create a node, give it one property. Then go back and randomly create relationships. I moved to the BatchInserter too. Is there a way to use more than one DB? Is it a file size limitation? Can I commit incrementally somehow? I looked at the BatchInterter source, and there doesn't seem to be anything that I can do. Exceptions and demo code below. Thanks, Todd Created 1516633 nodes in 33 seconds Exception in thread main org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write record[1419120] @[188742960] at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503) at org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311) at com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78) Caused by: java.io.IOException: Result too large at sun.nio.ch.FileDispatcher.pwrite0(Native Method) at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105) public class GraphIngestBatch { public static void main(String[] args) { try { // Create graph database service BatchInserter neo = new BatchInserterImpl(graphDB/, BatchInserterImpl.loadProperties(neo.props)); // Read raw text for name data FileReader namesFile = new FileReader(/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt); BufferedReader namesBuffer = new BufferedReader(namesFile); String name; // Start timer and counter Stopwatch timer = new Stopwatch(); timer.start(); long numberOfPeople = 1; // Initialize root node long person, neighbor; MapString,Object properties = new HashMapString,Object(); properties.put(name, Philberto Anderossono); neighbor = neo.createNode(properties); neo.createRelationship(neighbor, 0, DynamicRelationshipType.withName(ROOT), null); // Create people and neighbor relationships while ((name = namesBuffer.readLine()) != null) { properties.put(name, name); person = neo.createNode(properties); neo.createRelationship(person, neighbor, DynamicRelationshipType.withName(KNOWS), null); neo.createRelationship(neighbor, person, DynamicRelationshipType.withName(KNOWS), null); neighbor = person; numberOfPeople++; } namesBuffer.close(); namesFile.close(); System.out.println(Number of people is: + numberOfPeople); // Randomly generate relationships (minimum four per person + both neighbors, mean ten relationships / person) Long friend; int numberOfFriends = 1; for (person=1; personnumberOfPeople; person++) { for (int i=0; inumberOfFriends; i++)
Re: [Neo] Neo4J Scalability
Hi Todd, What OS do you run this on? The FileChannel.write fail with an exception saying buffer is too large is new to me. Try decrease the amount of memory you give the store files (if you have modified default configuration) and make sure you at least have disk space to write out 1/10th of the dedicated memory for each store file. I will run some tests on this code first thing in the morning to see if I can reproduce it. -Johan On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavishtoddstav...@gmail.com wrote: Hello All, I hope that I haven't been too demanding of this list. I am pushing a tight deadline. I can't seem to get Neo4J to scale above 1.5 million nodes. The code is simple, just create a node, give it one property. Then go back and randomly create relationships. I moved to the BatchInserter too. Is there a way to use more than one DB? Is it a file size limitation? Can I commit incrementally somehow? I looked at the BatchInterter source, and there doesn't seem to be anything that I can do. Exceptions and demo code below. Thanks, Todd Created 1516633 nodes in 33 seconds Exception in thread main org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write record[1419120] @[188742960] at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503) at org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311) at com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78) Caused by: java.io.IOException: Result too large at sun.nio.ch.FileDispatcher.pwrite0(Native Method) at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105) public class GraphIngestBatch { public static void main(String[] args) { try { // Create graph database service BatchInserter neo = new BatchInserterImpl(graphDB/, BatchInserterImpl.loadProperties(neo.props)); // Read raw text for name data FileReader namesFile = new FileReader(/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt); BufferedReader namesBuffer = new BufferedReader(namesFile); String name; // Start timer and counter Stopwatch timer = new Stopwatch(); timer.start(); long numberOfPeople = 1; // Initialize root node long person, neighbor; MapString,Object properties = new HashMapString,Object(); properties.put(name, Philberto Anderossono); neighbor = neo.createNode(properties); neo.createRelationship(neighbor, 0, DynamicRelationshipType.withName(ROOT), null); // Create people and neighbor relationships while ((name = namesBuffer.readLine()) != null) { properties.put(name, name); person = neo.createNode(properties); neo.createRelationship(person, neighbor, DynamicRelationshipType.withName(KNOWS), null); neo.createRelationship(neighbor, person, DynamicRelationshipType.withName(KNOWS), null); neighbor = person; numberOfPeople++; } namesBuffer.close(); namesFile.close(); System.out.println(Number of people is: + numberOfPeople); // Randomly generate relationships (minimum four per person + both neighbors, mean ten relationships / person)
Re: [Neo] Neo4J Scalability
He Johan, What OS do you run this on? The FileChannel.write fail with an exception saying buffer is too large is new to me. OSX. Snow Leopard. java -version java version 1.6.0_15 Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219) Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode) I can try it on centos too. I was going to try to run in it in EC2, but lost momentum and ideas to try. Try decrease the amount of memory you give the store files (if you have modified default configuration) and make sure you at least have disk space to write out 1/10th of the dedicated memory for each store file. Plenty of disk space. Do you mean pre-allocate the database file? How do you do that? I will run some tests on this code first thing in the morning to see if I can reproduce it. Thanks. I will send you the input files. And other code. Regards, Todd -Johan On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavishtoddstav...@gmail.com wrote: Hello All, I hope that I haven't been too demanding of this list. I am pushing a tight deadline. I can't seem to get Neo4J to scale above 1.5 million nodes. The code is simple, just create a node, give it one property. Then go back and randomly create relationships. I moved to the BatchInserter too. Is there a way to use more than one DB? Is it a file size limitation? Can I commit incrementally somehow? I looked at the BatchInterter source, and there doesn't seem to be anything that I can do. Exceptions and demo code below. Thanks, Todd Created 1516633 nodes in 33 seconds Exception in thread main org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write record[1419120] @[188742960] at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229) at org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503) at org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81) at org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500) at org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311) at com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78) Caused by: java.io.IOException: Result too large at sun.nio.ch.FileDispatcher.pwrite0(Native Method) at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651) at org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105) public class GraphIngestBatch { public static void main(String[] args) { try { // Create graph database service BatchInserter neo = new BatchInserterImpl(graphDB/, BatchInserterImpl.loadProperties(neo.props)); // Read raw text for name data FileReader namesFile = new FileReader(/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt); BufferedReader namesBuffer = new BufferedReader(namesFile); String name; // Start timer and counter Stopwatch timer = new Stopwatch(); timer.start(); long numberOfPeople = 1; // Initialize root node long person, neighbor; MapString,Object properties = new HashMapString,Object(); properties.put(name, Philberto Anderossono); neighbor = neo.createNode(properties); neo.createRelationship(neighbor, 0, DynamicRelationshipType.withName(ROOT), null); // Create people and neighbor relationships while ((name = namesBuffer.readLine()) != null) { properties.put(name, name); person = neo.createNode(properties); neo.createRelationship(person, neighbor, DynamicRelationshipType.withName(KNOWS), null); neo.createRelationship(neighbor, person, DynamicRelationshipType.withName(KNOWS), null);