Hi Michael, 2011/8/14 Michael Hunger <michael.hun...@neotechnology.com>: > Btw. even on Kei's Macbook it shouldn't take a second to add 1000 vertices. > Rather around a few milliseconds. Don't know what kind of overhead, the > blueprints / cytoscape managers > around the neo4j core add. I'm blueprints has a auto-tx mode where there is a > new tx per each operation and you have to disable that (if you use external > tx-managment).
I wrote a very simple tests using only Neo4j and Blueprints API. https://gist.github.com/1147365 The typical output (by iMac w/Core2Duo) is like the following: 1: With helper, add node transaction finished in 118 msec. 2: With helper, add node transaction finished in 84 msec. 3: With helper, add node transaction finished in 98 msec. 4: With helper, add node transaction finished in 57 msec. 5: With helper, add node transaction finished in 54 msec. 6: With helper, add node transaction finished in 22 msec. 7: With helper, add node transaction finished in 29 msec. 8: With helper, add node transaction finished in 21 msec. 9: With helper, add node transaction finished in 21 msec. 10: With helper, add node transaction finished in 72 msec. 1: Without helper, add node transaction finished in 15 msec. 2: Without helper, add node transaction finished in 5 msec. 3: Without helper, add node transaction finished in 7 msec. 4: Without helper, add node transaction finished in 7 msec. 5: Without helper, add node transaction finished in 8 msec. 6: Without helper, add node transaction finished in 4 msec. 7: Without helper, add node transaction finished in 4 msec. 8: Without helper, add node transaction finished in 4 msec. 9: Without helper, add node transaction finished in 3 msec. 10: Without helper, add node transaction finished in 3 msec. Does this looks normal? I think manual transaction is reasonably fast. Transactions with utility class are slower, but it's understandable. In any case, Apple's JVM does not have performance problem, but somehow it's slow on Oracle's JVM on Windows(?). I'll try the same tests on other machines. Thanks, Kei (The following is the message log from the test) Mon Aug 15 11:28:43 PDT 2011: Physical mem: 8192MB, Heap size: 123MB Mon Aug 15 11:28:43 PDT 2011: Kernel version: Neo4j - Graph Database Kernel 1.4.1 Mon Aug 15 11:28:43 PDT 2011: Neo4j - Graph Database Kernel 1.4.1 Mon Aug 15 11:28:43 PDT 2011: Operating System: Mac OS X; version: 10.6.8; arch: x86_64; cpus: 2 Mon Aug 15 11:28:43 PDT 2011: VM Name: Java HotSpot(TM) 64-Bit Server VM Mon Aug 15 11:28:43 PDT 2011: VM Vendor: Apple Inc. Mon Aug 15 11:28:43 PDT 2011: VM Version: 20.1-b02-384 Mon Aug 15 11:28:43 PDT 2011: Boot Class Path: Mon Aug 15 11:28:43 PDT 2011: Library Path: .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java Mon Aug 15 11:28:43 PDT 2011: Garbage Collector: ParNew: [Par Eden Space, Par Survivor Space] Mon Aug 15 11:28:43 PDT 2011: Garbage Collector: ConcurrentMarkSweep: [Par Eden Space, Par Survivor Space, CMS Old Gen, CMS Perm Gen] Mon Aug 15 11:28:43 PDT 2011: VM Arguments: [-Dfile.encoding=UTF-8] Mon Aug 15 11:28:43 PDT 2011: Mon Aug 15 11:28:43 PDT 2011: create=true Mon Aug 15 11:28:43 PDT 2011: logical_log=target/neo4jTestDB/nioneo_logical.log Mon Aug 15 11:28:43 PDT 2011: neo_store=target/neo4jTestDB/neostore Mon Aug 15 11:28:43 PDT 2011: neostore.nodestore.db.mapped_memory=20M Mon Aug 15 11:28:43 PDT 2011: neostore.propertystore.db.arrays.mapped_memory=130M Mon Aug 15 11:28:43 PDT 2011: neostore.propertystore.db.index.keys.mapped_memory=1M Mon Aug 15 11:28:43 PDT 2011: neostore.propertystore.db.index.mapped_memory=1M Mon Aug 15 11:28:43 PDT 2011: neostore.propertystore.db.mapped_memory=90M Mon Aug 15 11:28:43 PDT 2011: neostore.propertystore.db.strings.mapped_memory=130M Mon Aug 15 11:28:43 PDT 2011: neostore.relationshipstore.db.mapped_memory=100M Mon Aug 15 11:28:43 PDT 2011: node_auto_indexing=false Mon Aug 15 11:28:43 PDT 2011: rebuild_idgenerators_fast=true Mon Aug 15 11:28:43 PDT 2011: relationship_auto_indexing=false Mon Aug 15 11:28:43 PDT 2011: store_dir=target/neo4jTestDB Mon Aug 15 11:28:43 PDT 2011: use_memory_mapped_buffers=true > > Cheers > > Michael > > Am 14.08.2011 um 05:00 schrieb Michael Kirby: > >> Hello, >> >> Here is my messages.log , I also recently got a new computer and it takes >> about 3 seconds on that one. The heap for it 1809 MB. >> >> Thu Aug 11 12:05:17 PDT 2011: Opened [target\neo4jDB\nioneo_logical.log.1] >> clean empty log, version=0 >> Thu Aug 11 12:05:17 PDT 2011: Opened >> [C:\Users\Michael\blueprints-graph-cytoscapeMaster\target\neo4jDB\index\lucene.log.1] >> clean empty log, version=0 >> Thu Aug 11 12:05:17 PDT 2011: TM opening log: target\neo4jDB\tm_tx_log.2 >> Thu Aug 11 12:05:17 PDT 2011: --- CONFIGURATION START --- >> Thu Aug 11 12:05:17 PDT 2011: Physical mem: 3006MB, Heap size: 668MB >> Thu Aug 11 12:05:17 PDT 2011: Kernel version: Neo4j - Graph Database Kernel >> 1.4.1 >> Thu Aug 11 12:05:17 PDT 2011: Neo4j - Graph Database Kernel 1.4.1 >> Thu Aug 11 12:05:17 PDT 2011: Operating System: Windows 7; version: 6.1; >> arch: amd64; cpus: 2 >> Thu Aug 11 12:05:17 PDT 2011: VM Name: Java HotSpot(TM) 64-Bit Server VM >> Thu Aug 11 12:05:17 PDT 2011: VM Vendor: Sun Microsystems Inc. >> Thu Aug 11 12:05:17 PDT 2011: VM Version: 17.0-b17 >> Thu Aug 11 12:05:17 PDT 2011: Boot Class Path: >> C:\Java\jdk1.6.0_21\jre\lib\resources.jar;C:\Java\jdk1.6.0_21\jre\lib\rt.jar;C:\Java\jdk1.6.0_21\jre\lib\sunrsasign.jar;C:\Java\jdk1.6.0_21\jre\lib\jsse.jar;C:\Java\jdk1.6.0_21\jre\lib\jce.jar;C:\Java\jdk1.6.0_21\jre\lib\charsets.jar;C:\Java\jdk1.6.0_21\jre\classes >> Thu Aug 11 12:05:17 PDT 2011: Class Path: >> C:\Users\Michael\blueprints-graph-cytoscapeMaster\target\test-classes;C:\Users\Michael\blueprints-graph-cytoscapeMaster\target\classes;C:\Users\Michael\workspace\api-parent\model-api\target\classes;C:\Users\Michael\.m2\repository\org\cytoscape\event-api\3.0.0-alpha6-SNAPSHOT\event-api-3.0.0-alpha6-SNAPSHOT.jar;C:\Users\Michael\.m2\repository\org\cytoscape\work-api\3.0.0-alpha6-SNAPSHOT\work-api-3.0.0-alpha6-SNAPSHOT.jar;C:\Users\Michael\.m2\repository\org\cytoscape\wrappers\tinkerpop-blueprints-core\0.9\tinkerpop-blueprints-core-0.9.jar;C:\Users\Michael\.m2\repository\org\cytoscape\wrappers\tinkerpop-blueprints-neo4j-graph\0.9\tinkerpop-blueprints-neo4j-graph-0.9.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-kernel\1.4.1\neo4j-kernel-1.4.1.jar;C:\Users\Michael\.m2\repository\org\apache\geronimo\specs\geronimo-jta_1.1_spec\1.1.1\geronimo-jta_1.1_spec-1.1.1.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-lucene-index\1.4.1\neo4j-lucene-index-1. >> 4.1.jar;C:\Users\Michael\.m2\repository\org\apache\lucene\lucene-core\3.1.0\lucene-core-3.1.0.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-graph-algo\1.4.1\neo4j-graph-algo-1.4.1.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-udc\1.4.1\neo4j-udc-1.4.1.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-graph-matching\1.4.1\neo4j-graph-matching-1.4.1.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-cypher\1.4.1\neo4j-cypher-1.4.1.jar;C:\Users\Michael\.m2\repository\org\scala-lang\scala-library\2.9.0-1\scala-library-2.9.0-1.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-jmx\1.4.1\neo4j-jmx-1.4.1.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-kernel\1.4.1\neo4j-kernel-1.4.1-tests.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-ha\1.3\neo4j-ha-1.3.jar;C:\Users\Michael\.m2\repository\org\neo4j\neo4j-backup\1.3\neo4j-backup-1.3.jar;C:\Users\Michael\.m2\repository\org\apache\zookeeper\zookeeper\3.3.2\zookeeper-3.3.2.jar;C:\Users\Mi >> chael\.m2\repository\org\neo4j\neo4j-com\1.3\neo4j-com-1.3.jar;C:\Users\Michael\.m2\repository\org\apache\servicemix\bundles\org.apache.servicemix.bundles.netty\3.2.3.Final_1\org.apache.servicemix.bundles.netty-3.2.3.Final_1.jar;C:\Users\Michael\.m2\repository\log4j\log4j\1.2.16\log4j-1.2.16.jar;C:\Users\Michael\workspace\api-parent\model-api\target\test-classes;C:\Users\Michael\.m2\repository\org\cytoscape\event-api\3.0.0-alpha6-SNAPSHOT\event-api-3.0.0-alpha6-SNAPSHOT-tests.jar;C:\Users\Michael\.m2\repository\junit\junit\4.8.2\junit-4.8.2.jar;C:\Users\Michael\.m2\repository\org\mockito\mockito-all\1.8.5\mockito-all-1.8.5.jar;C:\Users\Michael\.m2\repository\org\cytoscape\wrappers\tinkerpop-blueprints-sail-graph\0.9\tinkerpop-blueprints-sail-graph-0.9.jar;C:\Users\Michael\.m2\repository\org\slf4j\slf4j-log4j12\1.5.6\slf4j-log4j12-1.5.6.jar;C:\Users\Michael\.m2\repository\org\slf4j\slf4j-api\1.5.6\slf4j-api-1.5.6.jar;C:\Users\Michael\.m2\repository\org\o >> penrdf\sesame\sesame-sail-api\2.4.0\sesame-sail-api-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-query\2.4.0\sesame-query-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-rio-api\2.4.0\sesame-rio-api-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-queryalgebra-model\2.4.0\sesame-queryalgebra-model-2.4.0.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-collections\2.8.0\aduna-commons-collections-2.8.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-model\2.4.0\sesame-model-2.4.0.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-i18n\1.4.0\aduna-commons-i18n-1.4.0.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-iteration\2.10.0\aduna-commons-iteration-2.10.0.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-concurrent\2.7.0\aduna-commons-concurrent-2.7.0.jar;C:\Users\Michael\.m2\repository\info\aduna\co >> mmons\aduna-commons-lang\2.9.0\aduna-commons-lang-2.9.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-queryparser-sparql\2.4.0\sesame-queryparser-sparql-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-queryparser-api\2.4.0\sesame-queryparser-api-2.4.0.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-net\2.7.0\aduna-commons-net-2.7.0.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-text\2.7.0\aduna-commons-text-2.7.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-sail-memory\2.4.0\sesame-sail-memory-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-sail-inferencer\2.4.0\sesame-sail-inferencer-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-queryparser-serql\2.4.0\sesame-queryparser-serql-2.4.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-queryalgebra-evaluation\2.4.0\sesame-queryalgebra-evaluation-2.4.0.jar;C:\ >> Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-io\2.10.0\aduna-commons-io-2.10.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-sail-nativerdf\2.4.0\sesame-sail-nativerdf-2.4.0.jar;C:\Users\Michael\.m2\repository\net\fortytwo\linked-data-sail\0.7\linked-data-sail-0.7.jar;C:\Users\Michael\.m2\repository\net\fortytwo\ripple\ripple-rdf\0.7\ripple-rdf-0.7.jar;C:\Users\Michael\.m2\repository\net\fortytwo\ripple\ripple-base\0.7\ripple-base-0.7.jar;C:\Users\Michael\.m2\repository\net\fortytwo\rdfflow\0.7\rdfflow-0.7.jar;C:\Users\Michael\.m2\repository\net\fortytwo\flow\0.7\flow-0.7.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-rio-n3\2.4.2\sesame-rio-n3-2.4.2.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-rio-rdfxml\2.4.2\sesame-rio-rdfxml-2.4.2.jar;C:\Users\Michael\.m2\repository\info\aduna\commons\aduna-commons-xml\2.7.0\aduna-commons-xml-2.7.0.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame >> \sesame-rio-trig\2.4.2\sesame-rio-trig-2.4.2.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-rio-trix\2.4.2\sesame-rio-trix-2.4.2.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-rio-turtle\2.4.2\sesame-rio-turtle-2.4.2.jar;C:\Users\Michael\.m2\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;C:\Users\Michael\.m2\repository\commons-logging\commons-logging\1.0.4\commons-logging-1.0.4.jar;C:\Users\Michael\.m2\repository\commons-codec\commons-codec\1.2\commons-codec-1.2.jar;C:\Users\Michael\.m2\repository\org\restlet\org.restlet\1.0.5\org.restlet-1.0.5.jar;C:\Users\Michael\.m2\repository\net\fortytwo\sesametools\nquads\1.4\nquads-1.4.jar;C:\Users\Michael\.m2\repository\org\openrdf\sesame\sesame-rio-ntriples\2.4.2\sesame-rio-ntriples-2.4.2.jar;C:\Users\Michael\.m2\repository\net\fortytwo\sesametools\common\1.4\common-1.4.jar;/C:/Users/Michael/.eclipse/org.eclipse.platform_3.6.1_1709980481/configurati >> on/org.eclipse.osgi/bundles/149/1/.cp/;/C:/Users/Michael/.eclipse/org.eclipse.platform_3.6.1_1709980481/configuration/org.eclipse.osgi/bundles/147/1/.cp/;/C:/Users/Michael/.eclipse/org.eclipse.platform_3.6.1_1709980481/configuration/org.eclipse.osgi/bundles/148/1/.cp/ >> Thu Aug 11 12:05:17 PDT 2011: Library Path: >> C:\Java\jdk1.6.0_21\bin;.;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program >> Files >> (x86)\PHP\;C:\Perl64\site\bin;C:\Perl64\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;c:\java\jdk1.6.0_21\bin\;C:\Program >> Files\SlikSvn\bin\ >> Thu Aug 11 12:05:17 PDT 2011: Garbage Collector: PS Scavenge: [PS Eden >> Space, PS Survivor Space] >> Thu Aug 11 12:05:17 PDT 2011: Garbage Collector: PS MarkSweep: [PS Eden >> Space, PS Survivor Space, PS Old Gen, PS Perm Gen] >> Thu Aug 11 12:05:17 PDT 2011: VM Arguments: [-Dfile.encoding=Cp1252] >> Thu Aug 11 12:05:17 PDT 2011: >> Thu Aug 11 12:05:17 PDT 2011: create=true >> Thu Aug 11 12:05:17 PDT 2011: logical_log=target\neo4jDB\nioneo_logical.log >> Thu Aug 11 12:05:17 PDT 2011: neo_store=target\neo4jDB\neostore >> Thu Aug 11 12:05:17 PDT 2011: neostore.nodestore.db.mapped_memory=20M >> Thu Aug 11 12:05:17 PDT 2011: >> neostore.propertystore.db.arrays.mapped_memory=130M >> Thu Aug 11 12:05:17 PDT 2011: >> neostore.propertystore.db.index.keys.mapped_memory=1M >> Thu Aug 11 12:05:17 PDT 2011: >> neostore.propertystore.db.index.mapped_memory=1M >> Thu Aug 11 12:05:17 PDT 2011: neostore.propertystore.db.mapped_memory=90M >> Thu Aug 11 12:05:17 PDT 2011: >> neostore.propertystore.db.strings.mapped_memory=130M >> Thu Aug 11 12:05:17 PDT 2011: >> neostore.relationshipstore.db.mapped_memory=100M >> Thu Aug 11 12:05:17 PDT 2011: node_auto_indexing=false >> Thu Aug 11 12:05:17 PDT 2011: rebuild_idgenerators_fast=true >> Thu Aug 11 12:05:17 PDT 2011: relationship_auto_indexing=false >> Thu Aug 11 12:05:17 PDT 2011: store_dir=target\neo4jDB >> Thu Aug 11 12:05:17 PDT 2011: use_memory_mapped_buffers=false >> Thu Aug 11 12:05:17 PDT 2011: --- CONFIGURATION END --- >> Thu Aug 11 12:05:17 PDT 2011: Extension >> org.neo4j.kernel.KernelExtension[kernel udc] loaded ok >> Thu Aug 11 12:05:17 PDT 2011: Extension >> org.neo4j.kernel.KernelExtension[kernel jmx] loaded ok >> Thu Aug 11 12:05:17 PDT 2011: Extension >> org.neo4j.kernel.KernelExtension[dummy] loaded ok >> Thu Aug 11 12:05:17 PDT 2011: Extension >> org.neo4j.kernel.KernelExtension[other dummy] loaded ok >> Thu Aug 11 12:05:17 PDT 2011: Extension >> org.neo4j.kernel.KernelExtension[online backup] loaded ok >> Thu Aug 11 12:11:20 PDT 2011: Closed log target\neo4jDB\nioneo_logical.log >> Thu Aug 11 12:11:22 PDT 2011: NeoStore closed >> Thu Aug 11 12:11:22 PDT 2011: Closed log >> C:\Users\Michael\blueprints-graph-cytoscapeMaster\target\neo4jDB\index\lucene.log >> Thu Aug 11 12:11:22 PDT 2011: NeoStore closed >> Thu Aug 11 12:11:22 PDT 2011: TM shutting down >> >> Thank you, >> >> Michael >> >> >> >> ---------- Forwarded message ---------- >> From: Michael Hunger <michael.hun...@neotechnology.com> >> Date: 2011/8/11 >> Subject: Re: [Neo4j] Very Slow Transaction only on Windows >> To: Neo4j user discussions <user@lists.neo4j.org> >> >> >> Kei, >> >> How much heap does he use for the test? >> >> On windows memory mapped files are allocated within the java heap >> space not outside as on *nix systems. >> >> There is a messages.log in the graphdb directory. >> >> Could you share its content with us? >> >> Thanks so much >> >> Michael >> >> Am 11.08.2011 um 21:58 schrieb Keiichiro Ono: >> >>> Hi. >>> I have a summer student working on Cytoscape (graph >>> visualization/analysis software)-Neo4j integration project using >>> Blueprints framework. >>> >>> https://github.com/keiono/blueprints-graph-cytoscape/wiki >>> >>> Currently, I've got a question from him which I do not know the answer. >>> >>> Here is the very simple test code. It adds 1000 Vertices to Neo4j >>> Graph using CommitManager: >>> >>> @Test >>> public void testPerformance() { >>> >>> long start = System.currentTimeMillis(); >>> final CommitManager manager = >>> TransactionalGraphHelper.createCommitManager( >>> (TransactionalGraph) graphImplementation, >>> 1000); >>> for (int i = 0; i < 1000; i++) { >>> net.addNode(); >>> manager.incrCounter(); >>> } >>> >>> manager.close(); >>> final long time = (System.currentTimeMillis() - start); >>> System.out.println("Nodes added in " + time + " msec."); >>> >>> assertEquals(1000, net.getNodeCount()); >>> >>> // If we does not use transaction utility method, this can be >>> minutes or hours... >>> assertTrue(2000>time); >>> } >>> >>> https://github.com/keiono/blueprints-graph-cytoscape/blob/master/src/test/java/org/cytoscape/blueprints/implementations/Neo4jTest.java >>> >>> where net is a wrapper object for Cytoscape graph model, and addNode() >>> method simply calles graph.addVertex() method in Blueprints' Graph >>> API. This test passes in less than a second on my MacBook, but on >>> Windows PC, it takes 1+ minutes to finish. Seems transaction is not >>> working correctly on Windows, but I do not understand why it happens >>> only on Windows. >>> >>> Could you give me any suggestions to solve this Windows-dependent problem? >>> >>> Thanks, >>> >>> Kei >>> Cytoscape Core Developer >>> >>> -- >>> Keiichiro Ono http://www.keiono.net/ >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >> >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> >> >> >> -- >> Keiichiro Ono http://www.keiono.net/ >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user > > -- Keiichiro Ono http://www.keiono.net/ _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
