While I can't share the code I can say that we have our our batch insert
process. It actually does searches first to determine if we are updating or
creating a given node. We run batch inserts of about 40-50k nodes pet tx. We
are loading several thousand of nodes per second. Overall times vary with the
complexity of the file but we have loaded files with over 4 million rows where
each row contains between 4 and 40 nodes in less than 12 hours
Sent from my iPhone
> On Aug 29, 2014, at 11:47, "'Curtis Mosters' via Neo4j"
> <neo4j@googlegroups.com> wrote:
>
> Okay but still that version is faster overall. Ok you get faster the GC
> error. That's right. Well compared to OrientDB it's very slow. Overall I can
> say that currently the inserting of really big files can only be done with
> your tool Michael.
>
> As I said will test this evening, hopefully working this time =)
>
> Am Freitag, 29. August 2014 15:16:05 UTC+2 schrieb Michael Hunger:
>>
>> Actually looking at your old version again you use just one single tx
>>
>> Repeated tx.success() don't have an effect
>>
>> Sent from mobile device
>>
>>> Am 29.08.2014 um 10:57 schrieb "'Curtis Mosters' via Neo4j"
>>> <ne...@googlegroups.com>:
>>>
>>> Tested in AWS on Intel Xeon CPU E5-2670 v2 @ 2,5 GHz.
>>>
>>> With the addition I need for 1 mio lines ~100 sec
>>>
>>> With my old version of inserting every line it was 14 sec.
>>>
>>> So somehow batch commiting is much slower.
>>>
>>> Am Donnerstag, 28. August 2014 22:18:46 UTC+2 schrieb Michael Hunger:
>>>>
>>>>
>>>>> Am 28.08.2014 um 19:17 schrieb 'Curtis Mosters' via Neo4j
>>>>> <ne...@googlegroups.com>:
>>>>>
>>>>> Hi I don't like the current speed of Neo4j with Java API.
>>>>>
>>>>> I have written an example that just inserts random data. In the case 160k:
>>>> Change it to batching commits every 30k to 50k nodes not every node.
>>>>
>>>>
>>>> Transaction tx = graphDb.beginTx();
>>>>
>>>>> for (int i = 0; i < 160000; i++) {
>>>>> myFirstNode = graphDb.createNode();
>>>>> myFirstNode.setProperty( "id", "1" );
>>>>> myFirstNode.setProperty( "name", "Duane Nickull, I
>>>>> Braineater" );
>>>> if (i % 30000 == 0) {
>>>>> tx.success();
>>>> tx.close();
>>>> tx = graphDb.beginTx();
>>>> }
>>>>> }
>>>>
>>>>> tx.success();
>>>> tx.close();
>>>>
>>>> With the java API you usually can import 50 to 100k nodes per second.
>>>>
>>>>
>>>>>
>>>>> this needs 22 seconds + ~2 minutes of tx.success() finishing or so.
>>>>> That's pretty bad if you ask me.
>>>>>
>>>>> So how can I improve the speed of the insert.
>>>>>
>>>>> Already thought about it, maybe with:
>>>>>
>>>>> disabling tx
>>>>> declare massive insert
>>>>> inserting every 10k
>>>>> So yeah basically help me to speed it up. Please don't say to use the
>>>>> batch-inserter. He is fine but noch practicable for me.
>>>>>
>>>>> Thank you.
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Neo4j" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to neo4j+un...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Neo4j" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to neo4j+un...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
