On Fri, Aug 26, 2011 at 2:46 PM, jadell <josh.ad...@gmail.com> wrote:
> Jim, > > Fair enough. For now, I'll just know not to try and make batches that big > :-) My own use case is for the transaction safety rather than trying to > create thousands of entities at once, so it doesn't effect me that much. I > just wanted to have something more concrete to tell other users who might > try. > > Thanks to all for helping me investigate! > Update on this: I did a bit of hacking on this just now, and was able to improve the memory usage quite a bit. The major culprits was jacksons deserializer (or rather, that we use their mapping serializer), and using a list of strings to store and eventually aggregate results rather than StringBuilder (which is my fault, since it's my code, and I have no clue why I did it like that). I just pushed this to master. My benchmarks look like this, showing number of inserts, each insert being a node and a property: Old: ------------------- 1000 : 1.340s 5000 : 3.013s 10000 : 4.304s 50000 : 18.120s 100000 : OutOfMemory 500000 : OutOfMemory New: ------------------- 1000 : 1.546s 5000 : 3.116s 10000 : 3.702s 50000 : 17.183s 100000 : 35.427s 500000 : OutOfMemory This is on a JVM with 1GB heap. The OutOfMemory that we see in the new setup is because I didn't go all the way and implement streaming output, so that would be cool to try as well. /Jake > -- Josh > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Neo4jPHP-batch-insert-benchmarks-tp3282984p3286721.html > Sent from the Neo4j Community Discussions mailing list archive at > Nabble.com. > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
