Hi Todd,
What OS do you run this on? The FileChannel.write fail with an
exception saying buffer is too large is new to me.
Try decrease the amount of memory you give the store files (if you
have modified default configuration) and make sure you at least have
disk space to write out 1/10th of the dedicated memory for each store
file.
I will run some tests on this code first thing in the morning to see
if I can reproduce it.
-Johan
On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavish<toddstav...@gmail.com> wrote:
> Hello All,
>
> I hope that I haven't been too demanding of this list. I am pushing a
> tight deadline. I can't seem to get Neo4J to scale above 1.5 million
> nodes. The code is simple, just create a node, give it one property.
> Then go back and randomly create relationships. I moved to the
> BatchInserter too. Is there a way to use more than one DB? Is it a
> file size limitation? Can I commit incrementally somehow? I looked at
> the BatchInterter source, and there doesn't seem to be anything that I
> can do.
>
> Exceptions and demo code below.
>
> Thanks,
> Todd
>
>
>
>
> Created 1516633 nodes in 33 seconds
>
> Exception in thread "main"
> org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write
> record[1419120] @[188742960]
> at
> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111)
> at
> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123)
> at
> org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229)
> at
> org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203)
> at
> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503)
> at
> org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106)
> at
> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
> at org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81)
> at
> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
> at
> org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311)
> at
> com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78)
> Caused by: java.io.IOException: Result too large
> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> at sun.nio.ch.IOUtil.write(IOUtil.java:75)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651)
> at
> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105)
>
>
> public class GraphIngestBatch
> {
>
> public static void main(String[] args)
> {
> try
> {
>
> // Create graph database service
> BatchInserter neo = new BatchInserterImpl("graphDB/",
> BatchInserterImpl.loadProperties("neo.props"));
>
> // Read raw text for name data
> FileReader namesFile = new
> FileReader("/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt");
> BufferedReader namesBuffer = new
> BufferedReader(namesFile);
> String name;
>
> // Start timer and counter
> Stopwatch timer = new Stopwatch();
> timer.start();
> long numberOfPeople = 1;
>
> // Initialize root node
> long person, neighbor;
> Map<String,Object> properties = new
> HashMap<String,Object>();
> properties.put("name", "Philberto Anderossono");
> neighbor = neo.createNode(properties);
> neo.createRelationship(neighbor, 0,
> DynamicRelationshipType.withName("ROOT"), null);
>
> // Create people and neighbor relationships
> while ((name = namesBuffer.readLine()) != null)
> {
> properties.put("name", name);
> person = neo.createNode(properties);
>
> neo.createRelationship(person, neighbor,
> DynamicRelationshipType.withName("KNOWS"), null);
> neo.createRelationship(neighbor, person,
> DynamicRelationshipType.withName("KNOWS"), null);
>
> neighbor = person;
> numberOfPeople++;
> }
> namesBuffer.close();
> namesFile.close();
>
> System.out.println("Number of people is: " +
> numberOfPeople);
>
> // Randomly generate relationships (minimum four per
> person + both
> neighbors, mean ten relationships / person)
> Long friend;
> int numberOfFriends = 1;
> for (person=1; person>numberOfPeople; person++)
> {
> for (int i=0; i<numberOfFriends; i++)
> {
> friend =
> Uniform.staticNextLongFromTo(1, numberOfPeople);
> if (person != friend)
> {
> neo.createRelationship(person,
> friend,
> DynamicRelationshipType.withName("KNOWS"), null);
> neo.createRelationship(friend,
> person,
> DynamicRelationshipType.withName("KNOWS"), null);
> }
> }
> }
>
> // Stop timer
> timer.stop();
> System.out.println("Created " + numberOfPeople + " people in
> " + timer.getElapsedTimeSecs() + " seconds");
>
> // shutdown, makes sure all changes are written to disk
> neo.shutdown();
>
> }
> catch (java.io.FileNotFoundException ex)
> {
> System.out.println("File does not exist.");
> }
> catch (java.io.IOException ex)
> {
> ex.printStackTrace() ;
> }
>
> }
> }
_______________________________________________
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
