Some more usage examples: http://hg.readify.net/neo4jclient/src/tip/Test/ApiUsageIdeas.cs
-- Tatham -----Original Message----- From: Tatham Oddie Sent: Tuesday, 27 September 2011 7:41 AM To: 'Peter Neubauer'; Neo4j user discussions Subject: .NET client for neo4j Hi Peter, We publish our client on NuGet: http://nuget.org/List/Packages/Neo4jClient (NuGet is .NET's package manager.) The source is at http://hg.readify.net/neo4jclient It supports: * all the basic CRUD operations * index management * Gremlin queries via a fluent interface: you use C# lambdas and get full code completion in the IDE as you type; we only compile it down to Gremlin and send it over the wire when you enumerate it or call a scalar method like .Count(). See http://dl.dropbox.com/u/5216304/FluentGremlin.png This week we moved to parametized Gremlin queries over the wire (eg, g.v(p0).out[[label:p1]]) and saw a good performance improvement in our downstream app. This change was implemented completely within Neo4jClient's query processor so downstream apps just got the improvement automatically. We've also started optimizing some of our internal methods to use batching. For example, creating, relating and indexing a node all happens in a single REST call. We'll probably end up exposing the batching API outside of the client too via some type of unit-of-work pattern. As for build, our CI triggers on a code push and publishes a new package to NuGet upon a successful build. If the tests pass, it gets published. (Our unit tests mock out all the REST calls so we have a high level of confidence in them. Risk is low anyway, as NuGet makes it easy to grab any previous version of the package.) -- Tatham -----Original Message----- From: neubauer.pe...@gmail.com [mailto:neubauer.pe...@gmail.com] On Behalf Of Peter Neubauer Sent: Tuesday, 27 September 2011 2:19 AM To: Neo4j user discussions Cc: Tatham Oddie Subject: Re: [Neo4j] Turning Off or Configuring log rotation Very cool, glad to hear things work out. Got any code for the .NET bindings? I think we should start pulling client libraries into our build system, so there are stable versions of the most used clients for Neo4j available as part of the QA process. WDYT? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Sep 26, 2011 at 1:24 PM, Romiko Derbynew <romiko.derby...@readify.net> wrote: > Hi Peter, > > Indeed our desired result is now achieved. Tatham added batching to our .net > client as well. We are more than happy to assist with HA as this is one of > the last hurdles we need to overcome. If you can let us know in advance when > the release is scheduled then we will incorporate it into our sprint for HA > featuring. > > Much appreciated and thx for 1.5 release, could not have timed it better. > > Sent from my iPhone > > On 26/09/2011, at 5:33 PM, "Peter Neubauer" > <peter.neuba...@neotechnology.com> wrote: > >> Romiko, >> soudns good. Is this solving your current problems or is there stuff >> left to take care of for the moment? Also, there will be some code >> coming out on deploying Neo4j HA on Azure. Would you be interested in >> trying that out and maybe take it a bit further? >> >> Cheers, >> >> /peter neubauer >> >> GTalk: neubauer.peter >> Skype peter.neubauer >> Phone +46 704 106975 >> LinkedIn http://www.linkedin.com/in/neubauer >> Twitter http://twitter.com/peterneubauer >> >> http://www.neo4j.org - Your high performance graph database. >> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >> >> >> >> On Mon, Sep 26, 2011 at 4:16 AM, Romiko Derbynew >> <romiko.derby...@readify.net> wrote: >>> Hi Guys, >>> >>> We managed to fix the performance issue. >>> >>> * Used Paramterised Queries - increased performance by 1 extra node >>> per second, reduced frequency usage of heap thus no jvm memory >>> errors >>> * Configured our .Net client to use 70 threads for rest calls (I chose this >>> number as Neo4j reserves 80, so I decided 70) 5 TIMES INCREASE IN >>> PERFORMANCE!!!! >>> >>> >>> We are now going to implement the batching to improve performance >>> even more :) >>> >>> I have updated the blog post with these findings. >>> http://romikoderbynew.com/2011/09/23/neo4j-java-jvm-garbage-collecti >>> on-profilingheap-sizelog-rotationlucene-rest-api-performance-analysi >>> s/ >>> >>> >>> I will write a separate post on the improvements from the batching. >>> >>> MyStory Welcome, Temporary from MyStory Administration. Logout? >>> Agency Import Job 301 >>> >>> Log output: >>> >>> [01:44:58] [Thread 10] Migration message received at: 26/09/2011 >>> 1:44:58 AM +00:00 >>> >>> [01:44:59] [Thread 10] Migration message payload: JobUniqueId: >>> 301, AgencyUniqueId: 201 >>> >>> [01:44:59] [Thread 10] Executing sql script: >>> e:\AzureTemp\Resources\4bb24cfd-0b35-4c91-83de-97f335159a49\temp\tem >>> p\RoleTemp\634526342992784034\output.sql >>> >>> [01:45:01] [Thread 10] Processing script command line 10000/37620 >>> >>> [01:45:03] [Thread 10] Processing script command line 20000/37620 >>> >>> [01:45:05] [Thread 10] Processing script command line 30000/37620 >>> >>> [01:45:06] [Thread 10] All sql script commands executed >>> 37620/37620 >>> >>> [01:45:13] [Thread 10] Importing 37619 clients into the database >>> >>> [01:45:44] [Thread 12] Processed 169/37619 clients, 5.62 clients >>> per second, 01:51:06 remaining >>> >>> [01:46:16] [Thread 29] Processed 360/37619 clients, 5.80 clients >>> per second, 01:47:02 remaining >>> >>> [01:46:29] [Thread 69] Processed 1000/37619 clients, 13.30 clients >>> per second, 00:45:52 remaining >>> >>> [01:46:50] [Thread 28] Processed 2000/37619 clients, 20.78 clients >>> per second, 00:28:34 remaining >>> >>> [01:47:11] [Thread 30] Processed 3000/37619 clients, 25.47 clients >>> per second, 00:22:39 remaining >>> >>> [01:47:33] [Thread 32] Processed 4000/37619 clients, 28.71 clients >>> per second, 00:19:31 remaining >>> >>> [01:47:55] [Thread 76] Processed 5000/37619 clients, 30.95 clients >>> per second, 00:17:33 remaining >>> >>> [01:48:18] [Thread 87] Processed 6000/37619 clients, 32.52 clients >>> per second, 00:16:12 remaining >>> >>> [01:48:46] [Thread 13] Processed 7000/37619 clients, 32.95 clients >>> per second, 00:15:29 remaining >>> >>> [01:49:10] [Thread 70] Processed 8000/37619 clients, 33.84 clients >>> per second, 00:14:35 remaining >>> >>> [01:49:36] [Thread 54] Processed 9000/37619 clients, 34.25 clients >>> per second, 00:13:55 remaining >>> >>> [01:50:02] [Thread 64] Processed 10000/37619 clients, 34.69 >>> clients per second, 00:13:16 remaining >>> >>> [01:50:29] [Thread 50] Processed 11000/37619 clients, 34.85 >>> clients per second, 00:12:43 remaining >>> >>> [01:50:57] [Thread 82] Processed 12000/37619 clients, 34.97 >>> clients per second, 00:12:12 remaining >>> >>> [01:51:25] [Thread 27] Processed 13000/37619 clients, 35.02 >>> clients per second, 00:11:42 remaining >>> >>> [01:51:54] [Thread 10] Processed 14000/37619 clients, 34.99 >>> clients per second, 00:11:15 remaining >>> >>> [01:52:24] [Thread 76] Processed 15000/37619 clients, 34.89 >>> clients per second, 00:10:48 remaining >>> >>> [01:52:54] [Thread 68] Processed 15970/37619 clients, 34.70 >>> clients per second, 00:10:23 remaining >>> >>> [01:52:55] [Thread 17] Processed 16000/37619 clients, 34.69 >>> clients per second, 00:10:23 remaining >>> >>> [01:53:25] [Thread 59] Processed 16918/37619 clients, 34.43 >>> clients per second, 00:10:01 remaining >>> >>> [01:53:27] [Thread 46] Processed 17000/37619 clients, 34.42 >>> clients per second, 00:09:59 remaining >>> >>> [01:53:58] [Thread 58] Processed 17998/37619 clients, 34.33 >>> clients per second, 00:09:31 remaining >>> >>> [01:53:58] [Thread 34] Processed 18000/37619 clients, 34.32 >>> clients per second, 00:09:31 remaining >>> >>> [01:54:28] [Thread 21] Processed 18900/37619 clients, 34.07 >>> clients per second, 00:09:09 remaining >>> >>> [01:54:34] [Thread 13] Processed 19000/37619 clients, 33.90 >>> clients per second, 00:09:09 remaining >>> >>> [01:55:04] [Thread 35] Processed 19973/37619 clients, 33.81 >>> clients per second, 00:08:41 remaining >>> >>> [01:55:05] [Thread 86] Processed 20000/37619 clients, 33.80 >>> clients per second, 00:08:41 remaining >>> >>> [01:55:36] [Thread 93] Processed 20910/37619 clients, 33.61 >>> clients per second, 00:08:17 remaining >>> >>> [01:55:40] [Thread 42] Processed 21000/37619 clients, 33.54 >>> clients per second, 00:08:15 remaining >>> >>> [01:56:10] [Thread 31] Processed 21886/37619 clients, 33.35 >>> clients per second, 00:07:51 remaining >>> >>> [01:56:14] [Thread 81] Processed 22000/37619 clients, 33.32 >>> clients per second, 00:07:48 remaining >>> >>> [01:56:44] [Thread 17] Processed 22809/37619 clients, 33.02 >>> clients per second, 00:07:28 remaining >>> >>> [01:56:51] [Thread 77] Processed 23000/37619 clients, 32.96 >>> clients per second, 00:07:23 remaining >>> >>> [01:57:22] [Thread 42] Processed 23770/37619 clients, 32.63 >>> clients per second, 00:07:04 remaining >>> >>> [01:57:30] [Thread 71] Processed 24000/37619 clients, 32.57 >>> clients per second, 00:06:58 remaining >>> >>> [01:58:01] [Thread 44] Processed 24652/37619 clients, 32.13 >>> clients per second, 00:06:43 remaining >>> >>> [01:58:16] [Thread 42] Processed 25000/37619 clients, 31.94 >>> clients per second, 00:06:35 remaining >>> >>> [01:58:47] [Thread 9] Processed 25691/37619 clients, 31.60 >>> clients per second, 00:06:17 remaining >>> >>> [01:59:03] [Thread 84] Processed 26000/37619 clients, 31.33 >>> clients per second, 00:06:10 remaining >>> >>> [01:59:36] [Thread 91] Processed 26625/37619 clients, 30.88 >>> clients per second, 00:05:56 remaining >>> >>> [01:59:57] [Thread 74] Processed 27000/37619 clients, 30.57 >>> clients per second, 00:05:47 remaining >>> >>> [02:00:27] [Thread 79] Processed 27583/37619 clients, 30.20 >>> clients per second, 00:05:32 remaining >>> >>> [02:00:47] [Thread 82] Processed 28000/37619 clients, 30.01 >>> clients per second, 00:05:20 remaining >>> >>> [02:01:17] [Thread 64] Processed 28591/37619 clients, 29.67 >>> clients per second, 00:05:04 remaining >>> >>> [02:01:38] [Thread 17] Processed 29000/37619 clients, 29.46 >>> clients per second, 00:04:52 remaining >>> >>> [02:02:08] [Thread 24] Processed 29610/37619 clients, 29.18 >>> clients per second, 00:04:34 remaining >>> >>> [02:02:29] [Thread 58] Processed 30000/37619 clients, 28.98 >>> clients per second, 00:04:22 remaining >>> >>> [02:02:59] [Thread 69] Processed 30566/37619 clients, 28.68 >>> clients per second, 00:04:05 remaining >>> >>> [02:03:23] [Thread 89] Processed 31000/37619 clients, 28.46 >>> clients per second, 00:03:52 remaining >>> >>> [02:03:23] [Thread 88] Processed 31000/37619 clients, 28.46 >>> clients per second, 00:03:52 remaining >>> >>> [02:03:53] [Thread 30] Processed 31556/37619 clients, 28.18 >>> clients per second, 00:03:35 remaining >>> >>> [02:04:16] [Thread 61] Processed 32000/37619 clients, 28.00 >>> clients per second, 00:03:20 remaining >>> >>> [02:04:48] [Thread 66] Processed 32568/37619 clients, 27.73 >>> clients per second, 00:03:02 remaining >>> >>> [02:05:18] [Thread 20] Processed 33000/37619 clients, 27.41 >>> clients per second, 00:02:48 remaining >>> >>> [02:05:48] [Thread 13] Processed 33529/37619 clients, 27.16 >>> clients per second, 00:02:30 remaining >>> >>> [02:06:15] [Thread 56] Processed 34000/37619 clients, 26.95 >>> clients per second, 00:02:14 remaining >>> >>> [02:06:45] [Thread 92] Processed 34442/37619 clients, 26.66 >>> clients per second, 00:01:59 remaining >>> >>> [02:07:16] [Thread 95] Processed 34959/37619 clients, 26.44 >>> clients per second, 00:01:40 remaining >>> >>> [02:07:17] [Thread 91] Processed 35000/37619 clients, 26.44 >>> clients per second, 00:01:39 remaining >>> >>> [02:07:49] [Thread 95] Processed 35369/37619 clients, 26.09 >>> clients per second, 00:01:26 remaining >>> >>> [02:08:19] [Thread 41] Processed 35867/37619 clients, 25.88 >>> clients per second, 00:01:07 remaining >>> >>> [02:08:28] [Thread 66] Processed 36000/37619 clients, 25.81 >>> clients per second, 00:01:02 remaining >>> >>> [02:08:59] [Thread 45] Processed 36485/37619 clients, 25.60 >>> clients per second, 00:00:44 remaining >>> >>> [02:09:29] [Thread 78] Processed 36922/37619 clients, 25.37 >>> clients per second, 00:00:27 remaining >>> >>> [02:09:34] [Thread 81] Processed 37000/37619 clients, 25.34 >>> clients per second, 00:00:24 remaining >>> >>> [02:10:04] [Thread 12] Processed 37287/37619 clients, 25.01 >>> clients per second, 00:00:13 remaining >>> >>> [02:10:27] [Thread 10] All 37619 clients imported >>> >>> >>> -----Original Message----- >>> From: user-boun...@lists.neo4j.org >>> [mailto:user-boun...@lists.neo4j.org] On Behalf Of Romiko Derbynew >>> Sent: Friday, 23 September 2011 1:06 PM >>> To: Neo4j user discussions >>> Cc: Tatham Oddie >>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>> >>> Apologise. >>> Issue Number: https://github.com/neo4j/community/issues/28 >>> Blog: http://bit.ly/r9hBZC >>> >>> Cheers >>> >>> -----Original Message----- >>> From: user-boun...@lists.neo4j.org >>> [mailto:user-boun...@lists.neo4j.org] On Behalf Of Romiko Derbynew >>> Sent: Friday, 23 September 2011 12:52 PM >>> To: Neo4j user discussions >>> Cc: Tatham Oddie >>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>> >>> Hi Peter, >>> >>> We solved the issue by disable index creations when importing 1000's of >>> nodes. So we will need to figure out why Lucene Index creations via the >>> rest api causes so much instability. Below is the history of work done. I >>> have also published a blog report on this here: >>> >>> >>> I have created an issue with this email as the content. Issue number >>> is: INSERT ISSUE NUMBER http://bit.ly/r9hBZC >>> >>> >>> Also the circuit breaker did not get triggered at all, which means the REST >>> API is responding with 100% uptime when just doing the insert >>> nodes/relationships. I guess we then have a performance issue with the >>> Lucene side. I am more than willing to try collect information on this for >>> you, if I enable FINEST log setting, will we be able to get details from >>> neo4j about the lucene issue and why it causes such problems? >>> >>> >>> Troubleshoot log: >>> >>> I managed to disable the tx logging via the console in the emebedded web >>> server, and then ran an import. >>> >>> * Each Node that gets inserted - does a query, insert node, insert >>> index entries >>> * At around 3000 node mark, the rest api became unresponsive >>> >>> I have optimised the JVM by: >>> Increases stack size to 2048k >>> Increase Heap size to 512m -> 1024m >>> Set JVM to Server Mode >>> The GC is now optimized (In GCViewer, very few second generaton collections >>> occur) Optmised the Gremlin code in the neo4jClient to used paramterised >>> queries. >>> Reduced the number concurrent threads from the .net application to 10 >>> instead of unlimited to the rest api. >>> Disabled LUCENE Indexes >>> >>> So with all the optimisation techniques from the performance guide being >>> followed, we still get a slow response. So I guess this is not the flush to >>> disk, as I notice my disk is very fast (SSD) and the %disk time and queue >>> length is very low. >>> >>> Primary Cause Of Performance Issue >>> Now, with the Lucene Index Calls via rest turned off, it is running >>> much faster!! :) >>> >>> So the issue for us was all the Lucene Index rest Api calls. I guess this >>> is the case as for each node we need to add +- 4-8 per node. This means, if >>> we can reduce lucene index calls as Tatham logged by parsing a dictionary >>> of key values in a json post, then we can reduce 4-8 rest calls to 1 call. >>> >>> So to CONFIRM, the REST API is STABLE when not creating the indexes >>> for each node we create. Note all the calls are multithreaded. See >>> below >>> >>> Log output: >>> MyStory Welcome, Temporary from MyStory Administration. Logout? >>> Agency Import Job 281 >>> >>> Log output: >>> >>> [02:07:36] [Thread 11] Migration message received at: 23/09/2011 >>> 2:07:36 AM +00:00 >>> >>> [02:07:36] [Thread 11] Migration message payload: JobUniqueId: 281, >>> AgencyUniqueId: 181 >>> >>> [02:07:37] [Thread 11] Executing sql script: >>> e:\AzureTemp\Resources\483c6036-8050-4387-a806-6e880d655b58\temp\tem >>> p\RoleTemp\634523764566141006\output.sql >>> >>> [02:07:39] [Thread 11] Processing script command line 10000/37620 >>> >>> [02:07:40] [Thread 11] Processing script command line 20000/37620 >>> >>> [02:07:42] [Thread 11] Processing script command line 30000/37620 >>> >>> [02:07:43] [Thread 11] All sql script commands executed 37620/37620 >>> >>> [02:07:51] [Thread 11] Importing 37619 clients into the database >>> >>> [02:08:22] [Thread 18] Processed 146/37619 clients, 4.72 clients >>> per second, 02:12:25 remaining >>> >>> [02:08:53] [Thread 35] Processed 302/37619 clients, 4.87 clients >>> per second, 02:07:43 remaining >>> >>> [02:09:24] [Thread 82] Processed 457/37619 clients, 4.92 clients >>> per second, 02:05:58 remaining >>> >>> [02:09:58] [Thread 55] Processed 624/37619 clients, 4.93 clients >>> per second, 02:05:02 remaining >>> >>> [02:10:29] [Thread 66] Processed 779/37619 clients, 4.93 clients >>> per second, 02:04:32 remaining >>> >>> [02:11:01] [Thread 36] Processed 933/37619 clients, 4.92 clients >>> per second, 02:04:12 remaining >>> >>> [02:11:13] [Thread 35] Processed 1000/37619 clients, 4.95 clients >>> per second, 02:03:21 remaining >>> >>> [02:11:14] [Thread 40] Processed 1000/37619 clients, 4.95 clients >>> per second, 02:03:21 remaining >>> >>> [02:11:48] [Thread 13] Processed 1147/37619 clients, 4.84 clients >>> per second, 02:05:40 remaining >>> >>> [02:12:21] [Thread 9] Processed 1291/37619 clients, 4.78 clients >>> per second, 02:06:40 remaining >>> >>> [02:12:52] [Thread 79] Processed 1437/37619 clients, 4.77 clients >>> per second, 02:06:24 remaining >>> >>> [02:13:23] [Thread 167] Processed 1565/37619 clients, 4.71 clients >>> per second, 02:07:33 remaining >>> >>> [02:13:56] [Thread 212] Processed 1708/37619 clients, 4.68 clients >>> per second, 02:08:00 remaining >>> >>> [02:14:27] [Thread 26] Processed 1839/37619 clients, 4.64 clients >>> per second, 02:08:31 remaining >>> >>> [02:14:58] [Thread 203] Processed 1971/37619 clients, 4.61 clients >>> per second, 02:08:50 remaining >>> >>> [02:15:02] [Thread 91] Processed 2000/37619 clients, 4.64 clients >>> per second, 02:08:04 remaining >>> >>> [02:15:04] [Thread 243] Processed 2000/37619 clients, 4.64 clients >>> per second, 02:08:04 remaining >>> >>> [02:15:06] [Thread 13] Processed 2000/37619 clients, 4.64 clients >>> per second, 02:08:04 remaining >>> >>> [02:15:09] [Thread 39] Processed 2000/37619 clients, 4.63 clients >>> per second, 02:08:04 remaining >>> >>> [02:15:42] [Thread 71] Processed 2161/37619 clients, 4.59 clients >>> per second, 02:08:47 remaining >>> >>> [02:16:15] [Thread 235] Processed 2305/37619 clients, 4.57 clients >>> per second, 02:08:45 remaining >>> >>> [02:16:48] [Thread 179] Processed 2448/37619 clients, 4.56 clients >>> per second, 02:08:34 remaining >>> >>> [02:17:20] [Thread 371] Processed 2591/37619 clients, 4.55 clients >>> per second, 02:08:11 remaining >>> >>> [02:17:50] [Thread 327] Processed 2725/37619 clients, 4.55 clients >>> per second, 02:07:52 remaining >>> >>> [02:18:23] [Thread 408] Processed 2865/37619 clients, 4.54 clients >>> per second, 02:07:43 remaining >>> >>> [02:18:52] [Thread 111] Processed 3000/37619 clients, 4.54 clients >>> per second, 02:07:04 remaining >>> >>> [02:18:53] [Thread 72] Processed 3000/37619 clients, 4.54 clients >>> per second, 02:07:04 remaining >>> >>> [02:18:56] [Thread 424] Processed 3000/37619 clients, 4.54 clients >>> per second, 02:07:05 remaining >>> >>> [02:18:58] [Thread 45] Processed 3000/37619 clients, 4.54 clients >>> per second, 02:07:05 remaining >>> >>> [02:19:00] [Thread 254] Processed 3000/37619 clients, 4.54 clients >>> per second, 02:07:05 remaining >>> >>> [02:19:02] [Thread 425] Processed 3000/37619 clients, 4.54 clients >>> per second, 02:07:05 remaining >>> >>> [02:19:36] [Thread 320] Processed 3207/37619 clients, 4.55 clients >>> per second, 02:06:02 remaining >>> >>> [02:20:06] [Thread 170] Processed 3350/37619 clients, 4.56 clients >>> per second, 02:05:18 remaining >>> >>> [02:20:39] [Thread 46] Processed 3506/37619 clients, 4.57 clients >>> per second, 02:04:31 remaining >>> >>> [02:21:11] [Thread 98] Processed 3638/37619 clients, 4.55 clients >>> per second, 02:04:27 remaining >>> >>> [02:21:44] [Thread 447] Processed 3783/37619 clients, 4.54 clients >>> per second, 02:04:11 remaining >>> >>> [02:22:18] [Thread 144] Processed 3927/37619 clients, 4.53 clients >>> per second, 02:03:56 remaining >>> >>> [02:22:34] [Thread 200] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:38 remaining >>> >>> [02:22:34] [Thread 398] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:38 remaining >>> >>> [02:22:36] [Thread 41] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:38 remaining >>> >>> [02:22:38] [Thread 470] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:38 remaining >>> >>> [02:22:39] [Thread 471] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:38 remaining >>> >>> [02:22:42] [Thread 335] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:38 remaining >>> >>> [02:22:44] [Thread 558] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:39 remaining >>> >>> [02:22:46] [Thread 557] Processed 4000/37619 clients, 4.53 clients >>> per second, 02:03:39 remaining >>> >>> [02:23:19] [Thread 587] Processed 4210/37619 clients, 4.54 clients >>> per second, 02:02:41 remaining >>> >>> [02:23:52] [Thread 159] Processed 4353/37619 clients, 4.53 clients >>> per second, 02:02:22 remaining >>> >>> [02:24:24] [Thread 311] Processed 4497/37619 clients, 4.53 clients >>> per second, 02:01:53 remaining >>> >>> [02:24:55] [Thread 229] Processed 4631/37619 clients, 4.52 clients >>> per second, 02:01:34 remaining >>> >>> [02:25:28] [Thread 200] Processed 4784/37619 clients, 4.53 clients >>> per second, 02:00:54 remaining >>> >>> [02:26:00] [Thread 245] Processed 4936/37619 clients, 4.53 clients >>> per second, 02:00:10 remaining >>> >>> [02:26:14] [Thread 693] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:53 remaining >>> >>> [02:26:15] [Thread 581] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:53 remaining >>> >>> [02:26:16] [Thread 694] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:53 remaining >>> >>> [02:26:18] [Thread 584] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:53 remaining >>> >>> [02:26:20] [Thread 416] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:53 remaining >>> >>> [02:26:22] [Thread 68] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:54 remaining >>> >>> [02:26:24] [Thread 46] Processed 5000/37619 clients, 4.53 clients >>> per second, 01:59:54 remaining >>> >>> [02:26:57] [Thread 431] Processed 5224/37619 clients, 4.56 clients >>> per second, 01:58:23 remaining >>> >>> [02:27:29] [Thread 627] Processed 5367/37619 clients, 4.56 clients >>> per second, 01:57:56 remaining >>> >>> [02:28:00] [Thread 766] Processed 5522/37619 clients, 4.57 clients >>> per second, 01:57:05 remaining >>> >>> [02:28:32] [Thread 659] Processed 5665/37619 clients, 4.57 clients >>> per second, 01:56:38 remaining >>> >>> [02:29:06] [Thread 32] Processed 5821/37619 clients, 4.57 clients >>> per second, 01:56:05 remaining >>> >>> [02:29:38] [Thread 693] Processed 5978/37619 clients, 4.57 clients >>> per second, 01:55:19 remaining >>> >>> [02:30:11] [Thread 358] Processed 6157/37619 clients, 4.59 clients >>> per second, 01:54:08 remaining >>> >>> [02:30:42] [Thread 615] Processed 6301/37619 clients, 4.59 clients >>> per second, 01:53:35 remaining >>> >>> [02:31:14] [Thread 631] Processed 6458/37619 clients, 4.60 clients >>> per second, 01:52:49 remaining >>> >>> [02:31:47] [Thread 229] Processed 6625/37619 clients, 4.61 clients >>> per second, 01:51:57 remaining >>> >>> [02:32:20] [Thread 663] Processed 6806/37619 clients, 4.63 clients >>> per second, 01:50:50 remaining >>> >>> [02:32:53] [Thread 202] Processed 6962/37619 clients, 4.63 clients >>> per second, 01:50:14 remaining >>> >>> [02:33:00] [Thread 965] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:00 remaining >>> >>> [02:33:03] [Thread 569] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:00 remaining >>> >>> [02:33:04] [Thread 481] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:01 remaining >>> >>> [02:33:06] [Thread 966] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:01 remaining >>> >>> [02:33:08] [Thread 342] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:01 remaining >>> >>> [02:33:11] [Thread 483] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:01 remaining >>> >>> [02:33:13] [Thread 484] Processed 7000/37619 clients, 4.64 clients >>> per second, 01:50:01 remaining >>> >>> [02:33:48] [Thread 68] Processed 7223/37619 clients, 4.64 clients >>> per second, 01:49:11 remaining >>> >>> [02:34:19] [Thread 131] Processed 7367/37619 clients, 4.64 clients >>> per second, 01:48:41 remaining >>> >>> [02:34:51] [Thread 869] Processed 7534/37619 clients, 4.65 clients >>> per second, 01:47:49 remaining >>> >>> [02:35:22] [Thread 376] Processed 7689/37619 clients, 4.66 clients >>> per second, 01:47:07 remaining >>> >>> [02:35:54] [Thread 1086] Processed 7857/37619 clients, 4.67 clients >>> per second, 01:46:16 remaining >>> >>> [02:36:27] [Thread 123] Processed 8048/37619 clients, 4.69 clients >>> per second, 01:45:06 remaining >>> >>> [02:37:01] [Thread 201] Processed 8227/37619 clients, 4.70 clients >>> per second, 01:44:11 remaining >>> >>> [02:37:33] [Thread 1161] Processed 8407/37619 clients, 4.72 clients >>> per second, 01:43:11 remaining >>> >>> [02:38:06] [Thread 59] Processed 8586/37619 clients, 4.73 clients >>> per second, 01:42:16 remaining >>> >>> [02:38:38] [Thread 300] Processed 8755/37619 clients, 4.74 clients >>> per second, 01:41:29 remaining >>> >>> [02:39:12] [Thread 862] Processed 8922/37619 clients, 4.74 clients >>> per second, 01:40:49 remaining >>> >>> [02:39:25] [Thread 1239] Processed 9000/37619 clients, 4.75 clients >>> per second, 01:40:21 remaining >>> >>> [02:39:26] [Thread 437] Processed 9000/37619 clients, 4.75 clients >>> per second, 01:40:21 remaining >>> >>> [02:39:28] [Thread 1240] Processed 9000/37619 clients, 4.75 clients >>> per second, 01:40:21 remaining >>> >>> [02:39:31] [Thread 23] Processed 9000/37619 clients, 4.75 clients >>> per second, 01:40:21 remaining >>> >>> [02:40:05] [Thread 888] Processed 9208/37619 clients, 4.76 clients >>> per second, 01:39:26 remaining >>> >>> [02:40:37] [Thread 320] Processed 9398/37619 clients, 4.78 clients >>> per second, 01:38:23 remaining >>> >>> [02:41:10] [Thread 921] Processed 9566/37619 clients, 4.79 clients >>> per second, 01:37:40 remaining >>> >>> [02:41:44] [Thread 792] Processed 9756/37619 clients, 4.80 clients >>> per second, 01:36:46 remaining >>> >>> [02:42:17] [Thread 958] Processed 9936/37619 clients, 4.81 clients >>> per second, 01:35:55 remaining >>> >>> [02:42:28] [Thread 1155] Processed 10000/37619 clients, 4.81 clients >>> per second, 01:35:37 remaining >>> >>> [02:42:30] [Thread 67] Processed 10000/37619 clients, 4.81 clients >>> per second, 01:35:37 remaining >>> >>> [02:42:33] [Thread 485] Processed 10000/37619 clients, 4.81 clients >>> per second, 01:35:37 remaining >>> >>> [02:42:35] [Thread 406] Processed 10000/37619 clients, 4.81 clients >>> per second, 01:35:37 remaining >>> >>> [02:42:37] [Thread 808] Processed 10000/37619 clients, 4.81 clients >>> per second, 01:35:37 remaining >>> >>> [02:42:39] [Thread 1378] Processed 10000/37619 clients, 4.81 clients >>> per second, 01:35:37 remaining >>> >>> [02:43:11] [Thread 1411] Processed 10234/37619 clients, 4.83 clients >>> per second, 01:34:34 remaining >>> >>> [02:43:45] [Thread 298] Processed 10391/37619 clients, 4.82 clients >>> per second, 01:34:03 remaining >>> >>> [02:44:16] [Thread 1459] Processed 10582/37619 clients, 4.84 clients >>> per second, 01:33:01 remaining >>> >>> [02:44:49] [Thread 1041] Processed 10761/37619 clients, 4.85 clients >>> per second, 01:32:16 remaining >>> >>> [02:45:22] [Thread 1057] Processed 10940/37619 clients, 4.86 clients >>> per second, 01:31:29 remaining >>> >>> [02:45:53] [Thread 753] Processed 11108/37619 clients, 4.87 clients >>> per second, 01:30:47 remaining >>> >>> [02:46:27] [Thread 910] Processed 11288/37619 clients, 4.87 clients >>> per second, 01:30:03 remaining >>> >>> [02:47:00] [Thread 1580] Processed 11477/37619 clients, 4.89 clients >>> per second, 01:29:10 remaining >>> >>> [02:47:32] [Thread 125] Processed 11645/37619 clients, 4.89 clients >>> per second, 01:28:30 remaining >>> >>> [02:48:06] [Thread 1628] Processed 11824/37619 clients, 4.90 clients >>> per second, 01:27:48 remaining >>> >>> [02:48:40] [Thread 1385] Processed 12003/37619 clients, 4.90 clients >>> per second, 01:27:06 remaining >>> >>> [02:49:12] [Thread 1401] Processed 12171/37619 clients, 4.91 clients >>> per second, 01:26:27 remaining >>> >>> [02:49:44] [Thread 1192] Processed 12331/37619 clients, 4.91 clients >>> per second, 01:25:53 remaining >>> >>> [02:50:15] [Thread 1444] Processed 12508/37619 clients, 4.92 clients >>> per second, 01:25:07 remaining >>> >>> [02:50:47] [Thread 1463] Processed 12684/37619 clients, 4.92 clients >>> per second, 01:24:24 remaining >>> >>> -----Original Message----- >>> From: user-boun...@lists.neo4j.org >>> [mailto:user-boun...@lists.neo4j.org] On Behalf Of Romiko Derbynew >>> Sent: Friday, 23 September 2011 9:37 AM >>> To: Neo4j user discussions >>> Cc: Tatham Oddie >>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>> >>> Hi Peter, >>> >>> Thank you so much for the tips, will look into this today and see >>> what I can come up with. After my musings will definitely file for >>> new features e.g >>> * Batching via rest api (Being on .Net we rely solely on the Rest >>> API) >>> * Configuration level control of transaction log default sizes e.g. >>> 10mb -> 100mb, disabled >>> * Explore ways to optimise flushing to disk without affecting the >>> rest api >>> >>> Regards >>> >>> -----Original Message----- >>> From: user-boun...@lists.neo4j.org >>> [mailto:user-boun...@lists.neo4j.org] On Behalf Of Peter Neubauer >>> Sent: Friday, 23 September 2011 7:28 AM >>> To: Neo4j user discussions >>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>> >>> Romiko, >>> Gremlin is basically executing Java on the server side and thus can do >>> anything java can do. >>> http://docs.neo4j.org/chunked/snapshot/gremlin-plugin.html#rest-api- >>> send-an-arbitrary-groovy-script---lucene-sorting >>> has an example of mixing native Neo4j API calls (g.createNode() etc) >>> after obtaining the injected Neo4j DB instance with >>> >>> g.getRawGraph(), in this case using Neo4j Index sorting. >>> >>> After that, just call the log rotation code. Let me know how it works out, >>> otherwise I can provide an example tomorrow? >>> >>> Cheers, >>> >>> /peter neubauer >>> >>> GTalk: neubauer.peter >>> Skype peter.neubauer >>> Phone +46 704 106975 >>> LinkedIn http://www.linkedin.com/in/neubauer >>> Twitter http://twitter.com/peterneubauer >>> >>> http://www.neo4j.org - Your high performance graph database. >>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>> >>> >>> >>> On Thu, Sep 22, 2011 at 11:17 PM, Romiko Derbynew >>> <romiko.derby...@readify.net> wrote: >>>> Hi Peter, >>>> >>>> I think this will be ok via gremlin. Have you got any sample gremlin code >>>> that can do this, that would be much appreciated. Also in the long term I >>>> think a combo of tx logs + rest API batching will be the solution. >>>> >>>> Sent from my iPhone >>>> >>>> On 22/09/2011, at 10:08 PM, "Peter Neubauer" >>>> <peter.neuba...@neotechnology.com> wrote: >>>> >>>>> Romiko, >>>>> I would suggest to do the TX log config via Gremlin for the time >>>>> being, so you can resolve it via scripting. For the exposure of >>>>> the Tx log rotate size as a configuration property (there are logs >>>>> for every XA resource participating in a transaction), please >>>>> raise an issue at https://github.com/neo4j/community/issues - good input! >>>>> >>>>> For the flushing to disk, we will merge a Deferred Write feature >>>>> back into master soonish (after 1.5) which lets the OS decide when >>>>> to flush, if you can stand the risk of a lost flush if things go down. >>>>> The REST interface is waiting for the TX to be finished, which is >>>>> why it is slowing down when you have big flushes. See >>>>> https://github.com/neo4j/community/issues/18 for the ticket on >>>>> this >>>>> :) >>>>> >>>>> Would that be ok? >>>>> >>>>> Cheers, >>>>> >>>>> /peter neubauer >>>>> >>>>> GTalk: neubauer.peter >>>>> Skype peter.neubauer >>>>> Phone +46 704 106975 >>>>> LinkedIn http://www.linkedin.com/in/neubauer >>>>> Twitter http://twitter.com/peterneubauer >>>>> >>>>> http://www.neo4j.org - Your high performance graph database. >>>>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>>>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>>>> >>>>> >>>>> >>>>> On Thu, Sep 22, 2011 at 1:52 PM, Romiko Derbynew >>>>> <romiko.derby...@readify.net> wrote: >>>>>> Hi, >>>>>> >>>>>> It's the database logging, I'm finding the transaction logs are causing >>>>>> the rest API to become unresponsive when it flushes to disk. Basically >>>>>> just want to turn it off or set the tlog size by configuration and not >>>>>> via code sample as it is demonstrated in the performance guide. >>>>>> >>>>>> The current work around without this setting is a circuit breaker >>>>>> pattern. But I am very concerned as the DB flush should not slow >>>>>> down further rest calls :( >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On 22/09/2011, at 6:36 PM, "Peter Neubauer" >>>>>> <peter.neuba...@neotechnology.com> wrote: >>>>>> >>>>>>> Romiko, >>>>>>> which log files are you referring to? For the database >>>>>>> transaction log rotation, look at the neo4j.properties file to >>>>>>> tune things, see >>>>>>> http://docs.neo4j.org/chunked/snapshot/server-configuration.html >>>>>>> >>>>>>> For the server, there is logging.properties to tweak stuff. Note >>>>>>> that these settings require a restart to take effect. >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> /peter neubauer >>>>>>> >>>>>>> GTalk: neubauer.peter >>>>>>> Skype peter.neubauer >>>>>>> Phone +46 704 106975 >>>>>>> LinkedIn http://www.linkedin.com/in/neubauer >>>>>>> Twitter http://twitter.com/peterneubauer >>>>>>> >>>>>>> http://www.neo4j.org - Your high performance graph >>>>>>> database. >>>>>>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>>>>>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 22, 2011 at 1:38 AM, Romiko Derbynew >>>>>>> <romiko.derby...@readify.net> wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using Neo4j on a Windows Server, is there any way I can configure >>>>>>>> log rotation via the configuration files? I would like to turn it off >>>>>>>> or try a different log rotation size. Since I am on windows, I am not >>>>>>>> sure how I would do it at runtime. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> _______________________________________________ >>>>>>>> Neo4j mailing list >>>>>>>> User@lists.neo4j.org >>>>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Neo4j mailing list >>>>>>> User@lists.neo4j.org >>>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>>> >>>>>> _______________________________________________ >>>>>> Neo4j mailing list >>>>>> User@lists.neo4j.org >>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>> >>>>> _______________________________________________ >>>>> Neo4j mailing list >>>>> User@lists.neo4j.org >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> User@lists.neo4j.org >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
