Hi Johan,
Thanks! Quick summary, it was definitely a config issue. Changing to
the recommend config settings makes the exceptions go away. Weird
though, those config values are for VM memory management. But the
exceptions looked like file I/O issues. How does Neo4J get around the
OS max file size limitation, does it just overflow into a new file?
> I successfully ran the code resulting in roughly 190M nodes/properties
> and 390M relationships injected in 35min (the condition in one of your
> for loops probably has a typo, should have been person<numberOfPeople
Oh thanks, I thought that I was accessing a nodeID that didn't exist,
so I tried to go above the total number of nodes. I forgot to change
it back. Sorry.
> I first tried on my laptop but ran out of disk space (with a similar
> exception you got) since the full load requires 41GB of free space.
> Could you please try the injection again (using similar configuration
> and verifying you have enough disk space)?
I had enough disk space, 200GB+. I am still getting heap issues, I
don't have enough memory on my laptop, but I will run it on the cloud
later and they should go away.
> Also (maybe not important since this is just a test but) relationships
> can be treated as bi-directional/non directed using the enum constant
> Direction.BOTH, so the two relationships a-KNOWS->b and b-KNOWS->a
> can be represented with one a-KNOWS-b.
This is just for traversals, no? You can't define it as bi-directional
on creation, can you? I noticed that Neo doesn't explicitly define
cardinality (many-to-many, one-to-many, etc). Everything is supported
though. And the API is concise, which is prefereable.
Thanks again.
Regards,
Todd
> Regards,
> -Johan
>
> On Wed, Sep 9, 2009 at 3:11 AM, Todd Stavish<toddstav...@gmail.com> wrote:
>> He Johan,
>>
>>> What OS do you run this on? The FileChannel.write fail with an
>>> exception saying buffer is too large is new to me.
>>
>> OSX. Snow Leopard.
>>
>> java -version
>> java version "1.6.0_15"
>> Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
>> Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
>>
>> I can try it on centos too. I was going to try to run in it in EC2,
>> but lost momentum and ideas to try.
>>
>>> Try decrease the amount of memory you give the store files (if you
>>> have modified default configuration) and make sure you at least have
>>> disk space to write out 1/10th of the dedicated memory for each store
>>> file.
>>
>> Plenty of disk space. Do you mean pre-allocate the database file? How
>> do you do that?
>>
>>> I will run some tests on this code first thing in the morning to see
>>> if I can reproduce it.
>>
>> Thanks. I will send you the input files. And other code.
>>
>> Regards,
>> Todd
>>
>>
>>> -Johan
>>>
>>> On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavish<toddstav...@gmail.com> wrote:
>>>> Hello All,
>>>>
>>>> I hope that I haven't been too demanding of this list. I am pushing a
>>>> tight deadline. I can't seem to get Neo4J to scale above 1.5 million
>>>> nodes. The code is simple, just create a node, give it one property.
>>>> Then go back and randomly create relationships. I moved to the
>>>> BatchInserter too. Is there a way to use more than one DB? Is it a
>>>> file size limitation? Can I commit incrementally somehow? I looked at
>>>> the BatchInterter source, and there doesn't seem to be anything that I
>>>> can do.
>>>>
>>>> Exceptions and demo code below.
>>>>
>>>> Thanks,
>>>> Todd
>>>>
>>>>
>>>>
>>>>
>>>> Created 1516633 nodes in 33 seconds
>>>>
>>>> Exception in thread "main"
>>>> org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write
>>>> record[1419120] @[188742960]
>>>> at
>>>> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111)
>>>> at
>>>> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123)
>>>> at
>>>> org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229)
>>>> at
>>>> org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203)
>>>> at
>>>> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503)
>>>> at
>>>> org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106)
>>>> at
>>>> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
>>>> at
>>>> org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81)
>>>> at
>>>> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
>>>> at
>>>> org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311)
>>>> at
>>>> com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78)
>>>> Caused by: java.io.IOException: Result too large
>>>> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
>>>> at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45)
>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:75)
>>>> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651)
>>>> at
>>>> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105)
>>>>
>>>>
>>>> public class GraphIngestBatch
>>>> {
>>>>
>>>> public static void main(String[] args)
>>>> {
>>>> try
>>>> {
>>>>
>>>> // Create graph database service
>>>> BatchInserter neo = new
>>>> BatchInserterImpl("graphDB/",
>>>> BatchInserterImpl.loadProperties("neo.props"));
>>>>
>>>> // Read raw text for name data
>>>> FileReader namesFile = new
>>>> FileReader("/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt");
>>>> BufferedReader namesBuffer = new
>>>> BufferedReader(namesFile);
>>>> String name;
>>>>
>>>> // Start timer and counter
>>>> Stopwatch timer = new Stopwatch();
>>>> timer.start();
>>>> long numberOfPeople = 1;
>>>>
>>>> // Initialize root node
>>>> long person, neighbor;
>>>> Map<String,Object> properties = new
>>>> HashMap<String,Object>();
>>>> properties.put("name", "Philberto Anderossono");
>>>> neighbor = neo.createNode(properties);
>>>> neo.createRelationship(neighbor, 0,
>>>> DynamicRelationshipType.withName("ROOT"), null);
>>>>
>>>> // Create people and neighbor relationships
>>>> while ((name = namesBuffer.readLine()) != null)
>>>> {
>>>> properties.put("name", name);
>>>> person = neo.createNode(properties);
>>>>
>>>> neo.createRelationship(person, neighbor,
>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>> neo.createRelationship(neighbor, person,
>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>>
>>>> neighbor = person;
>>>> numberOfPeople++;
>>>> }
>>>> namesBuffer.close();
>>>> namesFile.close();
>>>>
>>>> System.out.println("Number of people is: " +
>>>> numberOfPeople);
>>>>
>>>> // Randomly generate relationships (minimum four
>>>> per person + both
>>>> neighbors, mean ten relationships / person)
>>>> Long friend;
>>>> int numberOfFriends = 1;
>>>> for (person=1; person>numberOfPeople; person++)
>>>> {
>>>> for (int i=0; i<numberOfFriends; i++)
>>>> {
>>>> friend =
>>>> Uniform.staticNextLongFromTo(1, numberOfPeople);
>>>> if (person != friend)
>>>> {
>>>>
>>>> neo.createRelationship(person, friend,
>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>>
>>>> neo.createRelationship(friend, person,
>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>> }
>>>> }
>>>> }
>>>>
>>>> // Stop timer
>>>> timer.stop();
>>>> System.out.println("Created " + numberOfPeople + " people in
>>>> " + timer.getElapsedTimeSecs() + " seconds");
>>>>
>>>> // shutdown, makes sure all changes are written to
>>>> disk
>>>> neo.shutdown();
>>>>
>>>> }
>>>> catch (java.io.FileNotFoundException ex)
>>>> {
>>>> System.out.println("File does not exist.");
>>>> }
>>>> catch (java.io.IOException ex)
>>>> {
>>>> ex.printStackTrace() ;
>>>> }
>>>>
>>>> }
>>>> }
> _______________________________________________
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
_______________________________________________
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
