Hello All,
I hope that I haven't been too demanding of this list. I am pushing a
tight deadline. I can't seem to get Neo4J to scale above 1.5 million
nodes. The code is simple, just create a node, give it one property.
Then go back and randomly create relationships. I moved to the
BatchInserter too. Is there a way to use more than one DB? Is it a
file size limitation? Can I commit incrementally somehow? I looked at
the BatchInterter source, and there doesn't seem to be anything that I
can do.
Exceptions and demo code below.
Thanks,
Todd
Created 1516633 nodes in 33 seconds
Exception in thread "main"
org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write
record[1419120] @[188742960]
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111)
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123)
at
org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229)
at
org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503)
at
org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
at
org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311)
at com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78)
Caused by: java.io.IOException: Result too large
at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651)
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105)
public class GraphIngestBatch
{
public static void main(String[] args)
{
try
{
// Create graph database service
BatchInserter neo = new BatchInserterImpl("graphDB/",
BatchInserterImpl.loadProperties("neo.props"));
// Read raw text for name data
FileReader namesFile = new
FileReader("/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt");
BufferedReader namesBuffer = new
BufferedReader(namesFile);
String name;
// Start timer and counter
Stopwatch timer = new Stopwatch();
timer.start();
long numberOfPeople = 1;
// Initialize root node
long person, neighbor;
Map<String,Object> properties = new
HashMap<String,Object>();
properties.put("name", "Philberto Anderossono");
neighbor = neo.createNode(properties);
neo.createRelationship(neighbor, 0,
DynamicRelationshipType.withName("ROOT"), null);
// Create people and neighbor relationships
while ((name = namesBuffer.readLine()) != null)
{
properties.put("name", name);
person = neo.createNode(properties);
neo.createRelationship(person, neighbor,
DynamicRelationshipType.withName("KNOWS"), null);
neo.createRelationship(neighbor, person,
DynamicRelationshipType.withName("KNOWS"), null);
neighbor = person;
numberOfPeople++;
}
namesBuffer.close();
namesFile.close();
System.out.println("Number of people is: " +
numberOfPeople);
// Randomly generate relationships (minimum four per
person + both
neighbors, mean ten relationships / person)
Long friend;
int numberOfFriends = 1;
for (person=1; person>numberOfPeople; person++)
{
for (int i=0; i<numberOfFriends; i++)
{
friend =
Uniform.staticNextLongFromTo(1, numberOfPeople);
if (person != friend)
{
neo.createRelationship(person,
friend,
DynamicRelationshipType.withName("KNOWS"), null);
neo.createRelationship(friend,
person,
DynamicRelationshipType.withName("KNOWS"), null);
}
}
}
// Stop timer
timer.stop();
System.out.println("Created " + numberOfPeople + " people in
" + timer.getElapsedTimeSecs() + " seconds");
// shutdown, makes sure all changes are written to disk
neo.shutdown();
}
catch (java.io.FileNotFoundException ex)
{
System.out.println("File does not exist.");
}
catch (java.io.IOException ex)
{
ex.printStackTrace() ;
}
}
}
_______________________________________________
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
