The heap space stuff would make sense I think, because we currently deserialize and serialize in-place, keeping the whole thing in memory. Would be interesting to see if we could implement a setup that can stream the deserialization/serialization, getting rid of the memory overhead..
You said you are using out-of-the-box settings for the server, I don't remember off the top of my head what the default heap size is, but you might want to try giving it more RAM. I'm gonna guess that's where performance dies.. I'll have to look at what is proper HTTP behavior, but there should be a way we could start streaming back the response as it is being calculated, as long as we can come up with a good way of aborting if something fails.. Doing that would mean we don't have to keep a hundred thousand requests and responses in memory, which would completely change the performance situation. Big thanks for taking the time to put this together! /jake On Thu, Aug 25, 2011 at 3:52 PM, jadell <josh.ad...@gmail.com> wrote: > Hey Peter, > > I don't have any way of verifying on the server side, other than measuring > the time it takes for curl_exec to return a response. On the client side I > can see that PHP's json_encode/json_decode functions are taking less than > .5% of the total run time, even with a batch size of 10000. During one of > my > 100000 node attempts, I printed out the server response of the "500 Heap > space" error. It seemed like the last method in the stack trace was dealing > with a "Deserializer" class or method. I will try again and capture the > stack trace output to post here. > > Thanks, > > -- Josh Adell > > > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Neo4jPHP-batch-insert-benchmarks-tp3282984p3283926.html > Sent from the Neo4j Community Discussions mailing list archive at > Nabble.com. > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
