On Tue, Feb 1, 2011 at 10:19 PM, Tobias Ivarsson <tobias.ivars...@neotechnology.com> wrote:
> For getting a performance boost out of writes, doing multiple operations in > one transaction will give a much bigger gain than multiple threads though. > For your use case, I think two writer threads and a few hundred elements per > transaction is an appropriate size. I got some numbers. On a base of 10 rounds of 100000 rows each I got an average of 111.1811 sec to crunch a chunk of data, so it means that it would take 1.111811 ms to process a single row. A single file of data contains data for a single day and has an average of 2500000 rows so it would take approximately 46 minutes to crunch. The final db size is 588034K (574M) which has 1000000 rows so we can estimate that the final DB size would be 132307650K (126G). The current SQL DB is 60G and the app takes 4 and 1/2 hours to crunch a month of logs, identical machine. The test has been conducted starting from an empty db and the progress of chunk time is this one: 80307 ms, 83444 ms, 97162 ms, 131703 ms, 134647 ms, 104602 ms, 115944 ms, 112489 ms, 115660 ms, 135853 ms. There are 20 threads (default config) and they're processing 200 rows each within a single Transaction. The JVM is: OpenJDK Runtime Environment (build 1.6.0-b20) and was started with this options: -Djava.awt.headless=true -Xms40m -Xmx1536m -XX:+UseConcMarkSweepGC And ... What do you think? Thanks for any info you could give -- Massimo http://meridio.blogspot.com _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
