2009/9/10 Todd Stavish <toddstav...@gmail.com>:
> Hi Johan,
>
> Thanks! Quick summary, it was definitely a config issue. Changing to
> the recommend config settings makes the exceptions go away. Weird
> though, those config values are for VM memory management. But the
> exceptions looked like file I/O issues. How does Neo4J get around the
> OS max file size limitation, does it just overflow into a new file?
>
>> I successfully ran the code resulting in roughly 190M nodes/properties
>> and 390M relationships injected in 35min (the condition in one of your
>> for loops probably has a typo, should have been person<numberOfPeople
>
> Oh thanks, I thought that I was accessing a nodeID that didn't exist,
> so I tried to go above the total number of nodes. I forgot to change
> it back. Sorry.
>
>
>> I first tried on my laptop but ran out of disk space (with a similar
>> exception you got) since the full load requires 41GB of free space.
>> Could you please try the injection again (using similar configuration
>> and verifying you have enough disk space)?
>
> I had enough disk space, 200GB+. I am still getting heap issues, I
> don't have enough memory on my laptop, but I will run it on the cloud
> later and they should go away.
>
>> Also (maybe not important since this is just a test but) relationships
>> can be treated as bi-directional/non directed using the enum constant
>> Direction.BOTH, so the two relationships a-KNOWS->b and b-KNOWS->a
>> can be represented with one a-KNOWS-b.
>
> This is just for traversals, no? You can't define it as bi-directional
> on creation, can you? I noticed that Neo doesn't explicitly define
> cardinality (many-to-many, one-to-many, etc). Everything is supported
> though. And the API is concise, which is prefereable.
You can choose to disregard the direction in traversals and everywhere
else. For traversals use the Direction.BOTH enum and for getting
relationships for a Node you can use the node.getRelationships(
MY_RELATIONSHIP_TYPE ) method w/o the direction.
>
> Thanks again.
>
> Regards,
> Todd
>
>> Regards,
>> -Johan
>>
>> On Wed, Sep 9, 2009 at 3:11 AM, Todd Stavish<toddstav...@gmail.com> wrote:
>>> He Johan,
>>>
>>>> What OS do you run this on? The FileChannel.write fail with an
>>>> exception saying buffer is too large is new to me.
>>>
>>> OSX. Snow Leopard.
>>>
>>> java -version
>>> java version "1.6.0_15"
>>> Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
>>> Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
>>>
>>> I can try it on centos too. I was going to try to run in it in EC2,
>>> but lost momentum and ideas to try.
>>>
>>>> Try decrease the amount of memory you give the store files (if you
>>>> have modified default configuration) and make sure you at least have
>>>> disk space to write out 1/10th of the dedicated memory for each store
>>>> file.
>>>
>>> Plenty of disk space. Do you mean pre-allocate the database file? How
>>> do you do that?
>>>
>>>> I will run some tests on this code first thing in the morning to see
>>>> if I can reproduce it.
>>>
>>> Thanks. I will send you the input files. And other code.
>>>
>>> Regards,
>>> Todd
>>>
>>>
>>>> -Johan
>>>>
>>>> On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavish<toddstav...@gmail.com> wrote:
>>>>> Hello All,
>>>>>
>>>>> I hope that I haven't been too demanding of this list. I am pushing a
>>>>> tight deadline. I can't seem to get Neo4J to scale above 1.5 million
>>>>> nodes. The code is simple, just create a node, give it one property.
>>>>> Then go back and randomly create relationships. I moved to the
>>>>> BatchInserter too. Is there a way to use more than one DB? Is it a
>>>>> file size limitation? Can I commit incrementally somehow? I looked at
>>>>> the BatchInterter source, and there doesn't seem to be anything that I
>>>>> can do.
>>>>>
>>>>> Exceptions and demo code below.
>>>>>
>>>>> Thanks,
>>>>> Todd
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Created 1516633 nodes in 33 seconds
>>>>>
>>>>> Exception in thread "main"
>>>>> org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write
>>>>> record[1419120] @[188742960]
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
>>>>> at
>>>>> org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311)
>>>>> at
>>>>> com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78)
>>>>> Caused by: java.io.IOException: Result too large
>>>>> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
>>>>> at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45)
>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:75)
>>>>> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651)
>>>>> at
>>>>> org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105)
>>>>>
>>>>>
>>>>> public class GraphIngestBatch
>>>>> {
>>>>>
>>>>> public static void main(String[] args)
>>>>> {
>>>>> try
>>>>> {
>>>>>
>>>>> // Create graph database service
>>>>> BatchInserter neo = new
>>>>> BatchInserterImpl("graphDB/",
>>>>> BatchInserterImpl.loadProperties("neo.props"));
>>>>>
>>>>> // Read raw text for name data
>>>>> FileReader namesFile = new
>>>>> FileReader("/Users/todd/Code/pathfind/target/classes/rawData/lastNames.txt");
>>>>> BufferedReader namesBuffer = new
>>>>> BufferedReader(namesFile);
>>>>> String name;
>>>>>
>>>>> // Start timer and counter
>>>>> Stopwatch timer = new Stopwatch();
>>>>> timer.start();
>>>>> long numberOfPeople = 1;
>>>>>
>>>>> // Initialize root node
>>>>> long person, neighbor;
>>>>> Map<String,Object> properties = new
>>>>> HashMap<String,Object>();
>>>>> properties.put("name", "Philberto Anderossono");
>>>>> neighbor = neo.createNode(properties);
>>>>> neo.createRelationship(neighbor, 0,
>>>>> DynamicRelationshipType.withName("ROOT"), null);
>>>>>
>>>>> // Create people and neighbor relationships
>>>>> while ((name = namesBuffer.readLine()) != null)
>>>>> {
>>>>> properties.put("name", name);
>>>>> person = neo.createNode(properties);
>>>>>
>>>>> neo.createRelationship(person, neighbor,
>>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>>> neo.createRelationship(neighbor, person,
>>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>>>
>>>>> neighbor = person;
>>>>> numberOfPeople++;
>>>>> }
>>>>> namesBuffer.close();
>>>>> namesFile.close();
>>>>>
>>>>> System.out.println("Number of people is: " +
>>>>> numberOfPeople);
>>>>>
>>>>> // Randomly generate relationships (minimum four
>>>>> per person + both
>>>>> neighbors, mean ten relationships / person)
>>>>> Long friend;
>>>>> int numberOfFriends = 1;
>>>>> for (person=1; person>numberOfPeople; person++)
>>>>> {
>>>>> for (int i=0; i<numberOfFriends; i++)
>>>>> {
>>>>> friend =
>>>>> Uniform.staticNextLongFromTo(1, numberOfPeople);
>>>>> if (person != friend)
>>>>> {
>>>>>
>>>>> neo.createRelationship(person, friend,
>>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>>>
>>>>> neo.createRelationship(friend, person,
>>>>> DynamicRelationshipType.withName("KNOWS"), null);
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> // Stop timer
>>>>> timer.stop();
>>>>> System.out.println("Created " + numberOfPeople + " people
>>>>> in
>>>>> " + timer.getElapsedTimeSecs() + " seconds");
>>>>>
>>>>> // shutdown, makes sure all changes are written to
>>>>> disk
>>>>> neo.shutdown();
>>>>>
>>>>> }
>>>>> catch (java.io.FileNotFoundException ex)
>>>>> {
>>>>> System.out.println("File does not exist.");
>>>>> }
>>>>> catch (java.io.IOException ex)
>>>>> {
>>>>> ex.printStackTrace() ;
>>>>> }
>>>>>
>>>>> }
>>>>> }
>> _______________________________________________
>> Neo mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> _______________________________________________
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Mattias Persson, [matt...@neotechnology.com]
Neo Technology, www.neotechnology.com
_______________________________________________
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
