Romiko, soudns good. Is this solving your current problems or is there stuff left to take care of for the moment? Also, there will be some code coming out on deploying Neo4j HA on Azure. Would you be interested in trying that out and maybe take it a bit further?
Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Sep 26, 2011 at 4:16 AM, Romiko Derbynew <romiko.derby...@readify.net> wrote: > Hi Guys, > > We managed to fix the performance issue. > > * Used Paramterised Queries - increased performance by 1 extra node per > second, reduced frequency usage of heap thus no jvm memory errors > * Configured our .Net client to use 70 threads for rest calls (I chose this > number as Neo4j reserves 80, so I decided 70) 5 TIMES INCREASE IN > PERFORMANCE!!!! > > > We are now going to implement the batching to improve performance even more :) > > I have updated the blog post with these findings. > http://romikoderbynew.com/2011/09/23/neo4j-java-jvm-garbage-collection-profilingheap-sizelog-rotationlucene-rest-api-performance-analysis/ > > > I will write a separate post on the improvements from the batching. > > MyStory Welcome, Temporary from MyStory Administration. Logout? > Agency Import Job 301 > > Log output: > > [01:44:58] [Thread 10] Migration message received at: 26/09/2011 1:44:58 AM > +00:00 > > [01:44:59] [Thread 10] Migration message payload: JobUniqueId: 301, > AgencyUniqueId: 201 > > [01:44:59] [Thread 10] Executing sql script: > e:\AzureTemp\Resources\4bb24cfd-0b35-4c91-83de-97f335159a49\temp\temp\RoleTemp\634526342992784034\output.sql > > [01:45:01] [Thread 10] Processing script command line 10000/37620 > > [01:45:03] [Thread 10] Processing script command line 20000/37620 > > [01:45:05] [Thread 10] Processing script command line 30000/37620 > > [01:45:06] [Thread 10] All sql script commands executed 37620/37620 > > [01:45:13] [Thread 10] Importing 37619 clients into the database > > [01:45:44] [Thread 12] Processed 169/37619 clients, 5.62 clients per > second, 01:51:06 remaining > > [01:46:16] [Thread 29] Processed 360/37619 clients, 5.80 clients per > second, 01:47:02 remaining > > [01:46:29] [Thread 69] Processed 1000/37619 clients, 13.30 clients per > second, 00:45:52 remaining > > [01:46:50] [Thread 28] Processed 2000/37619 clients, 20.78 clients per > second, 00:28:34 remaining > > [01:47:11] [Thread 30] Processed 3000/37619 clients, 25.47 clients per > second, 00:22:39 remaining > > [01:47:33] [Thread 32] Processed 4000/37619 clients, 28.71 clients per > second, 00:19:31 remaining > > [01:47:55] [Thread 76] Processed 5000/37619 clients, 30.95 clients per > second, 00:17:33 remaining > > [01:48:18] [Thread 87] Processed 6000/37619 clients, 32.52 clients per > second, 00:16:12 remaining > > [01:48:46] [Thread 13] Processed 7000/37619 clients, 32.95 clients per > second, 00:15:29 remaining > > [01:49:10] [Thread 70] Processed 8000/37619 clients, 33.84 clients per > second, 00:14:35 remaining > > [01:49:36] [Thread 54] Processed 9000/37619 clients, 34.25 clients per > second, 00:13:55 remaining > > [01:50:02] [Thread 64] Processed 10000/37619 clients, 34.69 clients per > second, 00:13:16 remaining > > [01:50:29] [Thread 50] Processed 11000/37619 clients, 34.85 clients per > second, 00:12:43 remaining > > [01:50:57] [Thread 82] Processed 12000/37619 clients, 34.97 clients per > second, 00:12:12 remaining > > [01:51:25] [Thread 27] Processed 13000/37619 clients, 35.02 clients per > second, 00:11:42 remaining > > [01:51:54] [Thread 10] Processed 14000/37619 clients, 34.99 clients per > second, 00:11:15 remaining > > [01:52:24] [Thread 76] Processed 15000/37619 clients, 34.89 clients per > second, 00:10:48 remaining > > [01:52:54] [Thread 68] Processed 15970/37619 clients, 34.70 clients per > second, 00:10:23 remaining > > [01:52:55] [Thread 17] Processed 16000/37619 clients, 34.69 clients per > second, 00:10:23 remaining > > [01:53:25] [Thread 59] Processed 16918/37619 clients, 34.43 clients per > second, 00:10:01 remaining > > [01:53:27] [Thread 46] Processed 17000/37619 clients, 34.42 clients per > second, 00:09:59 remaining > > [01:53:58] [Thread 58] Processed 17998/37619 clients, 34.33 clients per > second, 00:09:31 remaining > > [01:53:58] [Thread 34] Processed 18000/37619 clients, 34.32 clients per > second, 00:09:31 remaining > > [01:54:28] [Thread 21] Processed 18900/37619 clients, 34.07 clients per > second, 00:09:09 remaining > > [01:54:34] [Thread 13] Processed 19000/37619 clients, 33.90 clients per > second, 00:09:09 remaining > > [01:55:04] [Thread 35] Processed 19973/37619 clients, 33.81 clients per > second, 00:08:41 remaining > > [01:55:05] [Thread 86] Processed 20000/37619 clients, 33.80 clients per > second, 00:08:41 remaining > > [01:55:36] [Thread 93] Processed 20910/37619 clients, 33.61 clients per > second, 00:08:17 remaining > > [01:55:40] [Thread 42] Processed 21000/37619 clients, 33.54 clients per > second, 00:08:15 remaining > > [01:56:10] [Thread 31] Processed 21886/37619 clients, 33.35 clients per > second, 00:07:51 remaining > > [01:56:14] [Thread 81] Processed 22000/37619 clients, 33.32 clients per > second, 00:07:48 remaining > > [01:56:44] [Thread 17] Processed 22809/37619 clients, 33.02 clients per > second, 00:07:28 remaining > > [01:56:51] [Thread 77] Processed 23000/37619 clients, 32.96 clients per > second, 00:07:23 remaining > > [01:57:22] [Thread 42] Processed 23770/37619 clients, 32.63 clients per > second, 00:07:04 remaining > > [01:57:30] [Thread 71] Processed 24000/37619 clients, 32.57 clients per > second, 00:06:58 remaining > > [01:58:01] [Thread 44] Processed 24652/37619 clients, 32.13 clients per > second, 00:06:43 remaining > > [01:58:16] [Thread 42] Processed 25000/37619 clients, 31.94 clients per > second, 00:06:35 remaining > > [01:58:47] [Thread 9] Processed 25691/37619 clients, 31.60 clients per > second, 00:06:17 remaining > > [01:59:03] [Thread 84] Processed 26000/37619 clients, 31.33 clients per > second, 00:06:10 remaining > > [01:59:36] [Thread 91] Processed 26625/37619 clients, 30.88 clients per > second, 00:05:56 remaining > > [01:59:57] [Thread 74] Processed 27000/37619 clients, 30.57 clients per > second, 00:05:47 remaining > > [02:00:27] [Thread 79] Processed 27583/37619 clients, 30.20 clients per > second, 00:05:32 remaining > > [02:00:47] [Thread 82] Processed 28000/37619 clients, 30.01 clients per > second, 00:05:20 remaining > > [02:01:17] [Thread 64] Processed 28591/37619 clients, 29.67 clients per > second, 00:05:04 remaining > > [02:01:38] [Thread 17] Processed 29000/37619 clients, 29.46 clients per > second, 00:04:52 remaining > > [02:02:08] [Thread 24] Processed 29610/37619 clients, 29.18 clients per > second, 00:04:34 remaining > > [02:02:29] [Thread 58] Processed 30000/37619 clients, 28.98 clients per > second, 00:04:22 remaining > > [02:02:59] [Thread 69] Processed 30566/37619 clients, 28.68 clients per > second, 00:04:05 remaining > > [02:03:23] [Thread 89] Processed 31000/37619 clients, 28.46 clients per > second, 00:03:52 remaining > > [02:03:23] [Thread 88] Processed 31000/37619 clients, 28.46 clients per > second, 00:03:52 remaining > > [02:03:53] [Thread 30] Processed 31556/37619 clients, 28.18 clients per > second, 00:03:35 remaining > > [02:04:16] [Thread 61] Processed 32000/37619 clients, 28.00 clients per > second, 00:03:20 remaining > > [02:04:48] [Thread 66] Processed 32568/37619 clients, 27.73 clients per > second, 00:03:02 remaining > > [02:05:18] [Thread 20] Processed 33000/37619 clients, 27.41 clients per > second, 00:02:48 remaining > > [02:05:48] [Thread 13] Processed 33529/37619 clients, 27.16 clients per > second, 00:02:30 remaining > > [02:06:15] [Thread 56] Processed 34000/37619 clients, 26.95 clients per > second, 00:02:14 remaining > > [02:06:45] [Thread 92] Processed 34442/37619 clients, 26.66 clients per > second, 00:01:59 remaining > > [02:07:16] [Thread 95] Processed 34959/37619 clients, 26.44 clients per > second, 00:01:40 remaining > > [02:07:17] [Thread 91] Processed 35000/37619 clients, 26.44 clients per > second, 00:01:39 remaining > > [02:07:49] [Thread 95] Processed 35369/37619 clients, 26.09 clients per > second, 00:01:26 remaining > > [02:08:19] [Thread 41] Processed 35867/37619 clients, 25.88 clients per > second, 00:01:07 remaining > > [02:08:28] [Thread 66] Processed 36000/37619 clients, 25.81 clients per > second, 00:01:02 remaining > > [02:08:59] [Thread 45] Processed 36485/37619 clients, 25.60 clients per > second, 00:00:44 remaining > > [02:09:29] [Thread 78] Processed 36922/37619 clients, 25.37 clients per > second, 00:00:27 remaining > > [02:09:34] [Thread 81] Processed 37000/37619 clients, 25.34 clients per > second, 00:00:24 remaining > > [02:10:04] [Thread 12] Processed 37287/37619 clients, 25.01 clients per > second, 00:00:13 remaining > > [02:10:27] [Thread 10] All 37619 clients imported > > > -----Original Message----- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On > Behalf Of Romiko Derbynew > Sent: Friday, 23 September 2011 1:06 PM > To: Neo4j user discussions > Cc: Tatham Oddie > Subject: Re: [Neo4j] Turning Off or Configuring log rotation > > Apologise. > Issue Number: https://github.com/neo4j/community/issues/28 > Blog: http://bit.ly/r9hBZC > > Cheers > > -----Original Message----- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On > Behalf Of Romiko Derbynew > Sent: Friday, 23 September 2011 12:52 PM > To: Neo4j user discussions > Cc: Tatham Oddie > Subject: Re: [Neo4j] Turning Off or Configuring log rotation > > Hi Peter, > > We solved the issue by disable index creations when importing 1000's of > nodes. So we will need to figure out why Lucene Index creations via the rest > api causes so much instability. Below is the history of work done. I have > also published a blog report on this here: > > > I have created an issue with this email as the content. Issue number is: > INSERT ISSUE NUMBER http://bit.ly/r9hBZC > > > Also the circuit breaker did not get triggered at all, which means the REST > API is responding with 100% uptime when just doing the insert > nodes/relationships. I guess we then have a performance issue with the Lucene > side. I am more than willing to try collect information on this for you, if I > enable FINEST log setting, will we be able to get details from neo4j about > the lucene issue and why it causes such problems? > > > Troubleshoot log: > > I managed to disable the tx logging via the console in the emebedded web > server, and then ran an import. > > * Each Node that gets inserted - does a query, insert node, insert index > entries > * At around 3000 node mark, the rest api became unresponsive > > I have optimised the JVM by: > Increases stack size to 2048k > Increase Heap size to 512m -> 1024m > Set JVM to Server Mode > The GC is now optimized (In GCViewer, very few second generaton collections > occur) Optmised the Gremlin code in the neo4jClient to used paramterised > queries. > Reduced the number concurrent threads from the .net application to 10 instead > of unlimited to the rest api. > Disabled LUCENE Indexes > > So with all the optimisation techniques from the performance guide being > followed, we still get a slow response. So I guess this is not the flush to > disk, as I notice my disk is very fast (SSD) and the %disk time and queue > length is very low. > > Primary Cause Of Performance Issue > Now, with the Lucene Index Calls via rest turned off, it is running much > faster!! :) > > So the issue for us was all the Lucene Index rest Api calls. I guess this is > the case as for each node we need to add +- 4-8 per node. This means, if we > can reduce lucene index calls as Tatham logged by parsing a dictionary of key > values in a json post, then we can reduce 4-8 rest calls to 1 call. > > So to CONFIRM, the REST API is STABLE when not creating the indexes for each > node we create. Note all the calls are multithreaded. See below > > Log output: > MyStory Welcome, Temporary from MyStory Administration. Logout? > Agency Import Job 281 > > Log output: > > [02:07:36] [Thread 11] Migration message received at: 23/09/2011 2:07:36 AM > +00:00 > > [02:07:36] [Thread 11] Migration message payload: JobUniqueId: 281, > AgencyUniqueId: 181 > > [02:07:37] [Thread 11] Executing sql script: > e:\AzureTemp\Resources\483c6036-8050-4387-a806-6e880d655b58\temp\temp\RoleTemp\634523764566141006\output.sql > > [02:07:39] [Thread 11] Processing script command line 10000/37620 > > [02:07:40] [Thread 11] Processing script command line 20000/37620 > > [02:07:42] [Thread 11] Processing script command line 30000/37620 > > [02:07:43] [Thread 11] All sql script commands executed 37620/37620 > > [02:07:51] [Thread 11] Importing 37619 clients into the database > > [02:08:22] [Thread 18] Processed 146/37619 clients, 4.72 clients per second, > 02:12:25 remaining > > [02:08:53] [Thread 35] Processed 302/37619 clients, 4.87 clients per second, > 02:07:43 remaining > > [02:09:24] [Thread 82] Processed 457/37619 clients, 4.92 clients per second, > 02:05:58 remaining > > [02:09:58] [Thread 55] Processed 624/37619 clients, 4.93 clients per second, > 02:05:02 remaining > > [02:10:29] [Thread 66] Processed 779/37619 clients, 4.93 clients per second, > 02:04:32 remaining > > [02:11:01] [Thread 36] Processed 933/37619 clients, 4.92 clients per second, > 02:04:12 remaining > > [02:11:13] [Thread 35] Processed 1000/37619 clients, 4.95 clients per > second, 02:03:21 remaining > > [02:11:14] [Thread 40] Processed 1000/37619 clients, 4.95 clients per > second, 02:03:21 remaining > > [02:11:48] [Thread 13] Processed 1147/37619 clients, 4.84 clients per > second, 02:05:40 remaining > > [02:12:21] [Thread 9] Processed 1291/37619 clients, 4.78 clients per > second, 02:06:40 remaining > > [02:12:52] [Thread 79] Processed 1437/37619 clients, 4.77 clients per > second, 02:06:24 remaining > > [02:13:23] [Thread 167] Processed 1565/37619 clients, 4.71 clients per > second, 02:07:33 remaining > > [02:13:56] [Thread 212] Processed 1708/37619 clients, 4.68 clients per > second, 02:08:00 remaining > > [02:14:27] [Thread 26] Processed 1839/37619 clients, 4.64 clients per > second, 02:08:31 remaining > > [02:14:58] [Thread 203] Processed 1971/37619 clients, 4.61 clients per > second, 02:08:50 remaining > > [02:15:02] [Thread 91] Processed 2000/37619 clients, 4.64 clients per > second, 02:08:04 remaining > > [02:15:04] [Thread 243] Processed 2000/37619 clients, 4.64 clients per > second, 02:08:04 remaining > > [02:15:06] [Thread 13] Processed 2000/37619 clients, 4.64 clients per > second, 02:08:04 remaining > > [02:15:09] [Thread 39] Processed 2000/37619 clients, 4.63 clients per > second, 02:08:04 remaining > > [02:15:42] [Thread 71] Processed 2161/37619 clients, 4.59 clients per > second, 02:08:47 remaining > > [02:16:15] [Thread 235] Processed 2305/37619 clients, 4.57 clients per > second, 02:08:45 remaining > > [02:16:48] [Thread 179] Processed 2448/37619 clients, 4.56 clients per > second, 02:08:34 remaining > > [02:17:20] [Thread 371] Processed 2591/37619 clients, 4.55 clients per > second, 02:08:11 remaining > > [02:17:50] [Thread 327] Processed 2725/37619 clients, 4.55 clients per > second, 02:07:52 remaining > > [02:18:23] [Thread 408] Processed 2865/37619 clients, 4.54 clients per > second, 02:07:43 remaining > > [02:18:52] [Thread 111] Processed 3000/37619 clients, 4.54 clients per > second, 02:07:04 remaining > > [02:18:53] [Thread 72] Processed 3000/37619 clients, 4.54 clients per > second, 02:07:04 remaining > > [02:18:56] [Thread 424] Processed 3000/37619 clients, 4.54 clients per > second, 02:07:05 remaining > > [02:18:58] [Thread 45] Processed 3000/37619 clients, 4.54 clients per > second, 02:07:05 remaining > > [02:19:00] [Thread 254] Processed 3000/37619 clients, 4.54 clients per > second, 02:07:05 remaining > > [02:19:02] [Thread 425] Processed 3000/37619 clients, 4.54 clients per > second, 02:07:05 remaining > > [02:19:36] [Thread 320] Processed 3207/37619 clients, 4.55 clients per > second, 02:06:02 remaining > > [02:20:06] [Thread 170] Processed 3350/37619 clients, 4.56 clients per > second, 02:05:18 remaining > > [02:20:39] [Thread 46] Processed 3506/37619 clients, 4.57 clients per > second, 02:04:31 remaining > > [02:21:11] [Thread 98] Processed 3638/37619 clients, 4.55 clients per > second, 02:04:27 remaining > > [02:21:44] [Thread 447] Processed 3783/37619 clients, 4.54 clients per > second, 02:04:11 remaining > > [02:22:18] [Thread 144] Processed 3927/37619 clients, 4.53 clients per > second, 02:03:56 remaining > > [02:22:34] [Thread 200] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:38 remaining > > [02:22:34] [Thread 398] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:38 remaining > > [02:22:36] [Thread 41] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:38 remaining > > [02:22:38] [Thread 470] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:38 remaining > > [02:22:39] [Thread 471] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:38 remaining > > [02:22:42] [Thread 335] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:38 remaining > > [02:22:44] [Thread 558] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:39 remaining > > [02:22:46] [Thread 557] Processed 4000/37619 clients, 4.53 clients per > second, 02:03:39 remaining > > [02:23:19] [Thread 587] Processed 4210/37619 clients, 4.54 clients per > second, 02:02:41 remaining > > [02:23:52] [Thread 159] Processed 4353/37619 clients, 4.53 clients per > second, 02:02:22 remaining > > [02:24:24] [Thread 311] Processed 4497/37619 clients, 4.53 clients per > second, 02:01:53 remaining > > [02:24:55] [Thread 229] Processed 4631/37619 clients, 4.52 clients per > second, 02:01:34 remaining > > [02:25:28] [Thread 200] Processed 4784/37619 clients, 4.53 clients per > second, 02:00:54 remaining > > [02:26:00] [Thread 245] Processed 4936/37619 clients, 4.53 clients per > second, 02:00:10 remaining > > [02:26:14] [Thread 693] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:53 remaining > > [02:26:15] [Thread 581] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:53 remaining > > [02:26:16] [Thread 694] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:53 remaining > > [02:26:18] [Thread 584] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:53 remaining > > [02:26:20] [Thread 416] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:53 remaining > > [02:26:22] [Thread 68] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:54 remaining > > [02:26:24] [Thread 46] Processed 5000/37619 clients, 4.53 clients per > second, 01:59:54 remaining > > [02:26:57] [Thread 431] Processed 5224/37619 clients, 4.56 clients per > second, 01:58:23 remaining > > [02:27:29] [Thread 627] Processed 5367/37619 clients, 4.56 clients per > second, 01:57:56 remaining > > [02:28:00] [Thread 766] Processed 5522/37619 clients, 4.57 clients per > second, 01:57:05 remaining > > [02:28:32] [Thread 659] Processed 5665/37619 clients, 4.57 clients per > second, 01:56:38 remaining > > [02:29:06] [Thread 32] Processed 5821/37619 clients, 4.57 clients per > second, 01:56:05 remaining > > [02:29:38] [Thread 693] Processed 5978/37619 clients, 4.57 clients per > second, 01:55:19 remaining > > [02:30:11] [Thread 358] Processed 6157/37619 clients, 4.59 clients per > second, 01:54:08 remaining > > [02:30:42] [Thread 615] Processed 6301/37619 clients, 4.59 clients per > second, 01:53:35 remaining > > [02:31:14] [Thread 631] Processed 6458/37619 clients, 4.60 clients per > second, 01:52:49 remaining > > [02:31:47] [Thread 229] Processed 6625/37619 clients, 4.61 clients per > second, 01:51:57 remaining > > [02:32:20] [Thread 663] Processed 6806/37619 clients, 4.63 clients per > second, 01:50:50 remaining > > [02:32:53] [Thread 202] Processed 6962/37619 clients, 4.63 clients per > second, 01:50:14 remaining > > [02:33:00] [Thread 965] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:00 remaining > > [02:33:03] [Thread 569] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:00 remaining > > [02:33:04] [Thread 481] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:01 remaining > > [02:33:06] [Thread 966] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:01 remaining > > [02:33:08] [Thread 342] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:01 remaining > > [02:33:11] [Thread 483] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:01 remaining > > [02:33:13] [Thread 484] Processed 7000/37619 clients, 4.64 clients per > second, 01:50:01 remaining > > [02:33:48] [Thread 68] Processed 7223/37619 clients, 4.64 clients per > second, 01:49:11 remaining > > [02:34:19] [Thread 131] Processed 7367/37619 clients, 4.64 clients per > second, 01:48:41 remaining > > [02:34:51] [Thread 869] Processed 7534/37619 clients, 4.65 clients per > second, 01:47:49 remaining > > [02:35:22] [Thread 376] Processed 7689/37619 clients, 4.66 clients per > second, 01:47:07 remaining > > [02:35:54] [Thread 1086] Processed 7857/37619 clients, 4.67 clients per > second, 01:46:16 remaining > > [02:36:27] [Thread 123] Processed 8048/37619 clients, 4.69 clients per > second, 01:45:06 remaining > > [02:37:01] [Thread 201] Processed 8227/37619 clients, 4.70 clients per > second, 01:44:11 remaining > > [02:37:33] [Thread 1161] Processed 8407/37619 clients, 4.72 clients per > second, 01:43:11 remaining > > [02:38:06] [Thread 59] Processed 8586/37619 clients, 4.73 clients per > second, 01:42:16 remaining > > [02:38:38] [Thread 300] Processed 8755/37619 clients, 4.74 clients per > second, 01:41:29 remaining > > [02:39:12] [Thread 862] Processed 8922/37619 clients, 4.74 clients per > second, 01:40:49 remaining > > [02:39:25] [Thread 1239] Processed 9000/37619 clients, 4.75 clients per > second, 01:40:21 remaining > > [02:39:26] [Thread 437] Processed 9000/37619 clients, 4.75 clients per > second, 01:40:21 remaining > > [02:39:28] [Thread 1240] Processed 9000/37619 clients, 4.75 clients per > second, 01:40:21 remaining > > [02:39:31] [Thread 23] Processed 9000/37619 clients, 4.75 clients per > second, 01:40:21 remaining > > [02:40:05] [Thread 888] Processed 9208/37619 clients, 4.76 clients per > second, 01:39:26 remaining > > [02:40:37] [Thread 320] Processed 9398/37619 clients, 4.78 clients per > second, 01:38:23 remaining > > [02:41:10] [Thread 921] Processed 9566/37619 clients, 4.79 clients per > second, 01:37:40 remaining > > [02:41:44] [Thread 792] Processed 9756/37619 clients, 4.80 clients per > second, 01:36:46 remaining > > [02:42:17] [Thread 958] Processed 9936/37619 clients, 4.81 clients per > second, 01:35:55 remaining > > [02:42:28] [Thread 1155] Processed 10000/37619 clients, 4.81 clients per > second, 01:35:37 remaining > > [02:42:30] [Thread 67] Processed 10000/37619 clients, 4.81 clients per > second, 01:35:37 remaining > > [02:42:33] [Thread 485] Processed 10000/37619 clients, 4.81 clients per > second, 01:35:37 remaining > > [02:42:35] [Thread 406] Processed 10000/37619 clients, 4.81 clients per > second, 01:35:37 remaining > > [02:42:37] [Thread 808] Processed 10000/37619 clients, 4.81 clients per > second, 01:35:37 remaining > > [02:42:39] [Thread 1378] Processed 10000/37619 clients, 4.81 clients per > second, 01:35:37 remaining > > [02:43:11] [Thread 1411] Processed 10234/37619 clients, 4.83 clients per > second, 01:34:34 remaining > > [02:43:45] [Thread 298] Processed 10391/37619 clients, 4.82 clients per > second, 01:34:03 remaining > > [02:44:16] [Thread 1459] Processed 10582/37619 clients, 4.84 clients per > second, 01:33:01 remaining > > [02:44:49] [Thread 1041] Processed 10761/37619 clients, 4.85 clients per > second, 01:32:16 remaining > > [02:45:22] [Thread 1057] Processed 10940/37619 clients, 4.86 clients per > second, 01:31:29 remaining > > [02:45:53] [Thread 753] Processed 11108/37619 clients, 4.87 clients per > second, 01:30:47 remaining > > [02:46:27] [Thread 910] Processed 11288/37619 clients, 4.87 clients per > second, 01:30:03 remaining > > [02:47:00] [Thread 1580] Processed 11477/37619 clients, 4.89 clients per > second, 01:29:10 remaining > > [02:47:32] [Thread 125] Processed 11645/37619 clients, 4.89 clients per > second, 01:28:30 remaining > > [02:48:06] [Thread 1628] Processed 11824/37619 clients, 4.90 clients per > second, 01:27:48 remaining > > [02:48:40] [Thread 1385] Processed 12003/37619 clients, 4.90 clients per > second, 01:27:06 remaining > > [02:49:12] [Thread 1401] Processed 12171/37619 clients, 4.91 clients per > second, 01:26:27 remaining > > [02:49:44] [Thread 1192] Processed 12331/37619 clients, 4.91 clients per > second, 01:25:53 remaining > > [02:50:15] [Thread 1444] Processed 12508/37619 clients, 4.92 clients per > second, 01:25:07 remaining > > [02:50:47] [Thread 1463] Processed 12684/37619 clients, 4.92 clients per > second, 01:24:24 remaining > > -----Original Message----- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On > Behalf Of Romiko Derbynew > Sent: Friday, 23 September 2011 9:37 AM > To: Neo4j user discussions > Cc: Tatham Oddie > Subject: Re: [Neo4j] Turning Off or Configuring log rotation > > Hi Peter, > > Thank you so much for the tips, will look into this today and see what I can > come up with. After my musings will definitely file for new features e.g > * Batching via rest api (Being on .Net we rely solely on the Rest API) > * Configuration level control of transaction log default sizes e.g. 10mb -> > 100mb, disabled > * Explore ways to optimise flushing to disk without affecting the rest api > > Regards > > -----Original Message----- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On > Behalf Of Peter Neubauer > Sent: Friday, 23 September 2011 7:28 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] Turning Off or Configuring log rotation > > Romiko, > Gremlin is basically executing Java on the server side and thus can do > anything java can do. > http://docs.neo4j.org/chunked/snapshot/gremlin-plugin.html#rest-api-send-an-arbitrary-groovy-script---lucene-sorting > has an example of mixing native Neo4j API calls (g.createNode() etc) after > obtaining the injected Neo4j DB instance with > > g.getRawGraph(), in this case using Neo4j Index sorting. > > After that, just call the log rotation code. Let me know how it works out, > otherwise I can provide an example tomorrow? > > Cheers, > > /peter neubauer > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://startupbootcamp.org/ - Öresund - Innovation happens HERE. > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. > > > > On Thu, Sep 22, 2011 at 11:17 PM, Romiko Derbynew > <romiko.derby...@readify.net> wrote: >> Hi Peter, >> >> I think this will be ok via gremlin. Have you got any sample gremlin code >> that can do this, that would be much appreciated. Also in the long term I >> think a combo of tx logs + rest API batching will be the solution. >> >> Sent from my iPhone >> >> On 22/09/2011, at 10:08 PM, "Peter Neubauer" >> <peter.neuba...@neotechnology.com> wrote: >> >>> Romiko, >>> I would suggest to do the TX log config via Gremlin for the time >>> being, so you can resolve it via scripting. For the exposure of the >>> Tx log rotate size as a configuration property (there are logs for >>> every XA resource participating in a transaction), please raise an >>> issue at https://github.com/neo4j/community/issues - good input! >>> >>> For the flushing to disk, we will merge a Deferred Write feature back >>> into master soonish (after 1.5) which lets the OS decide when to >>> flush, if you can stand the risk of a lost flush if things go down. >>> The REST interface is waiting for the TX to be finished, which is why >>> it is slowing down when you have big flushes. See >>> https://github.com/neo4j/community/issues/18 for the ticket on this >>> :) >>> >>> Would that be ok? >>> >>> Cheers, >>> >>> /peter neubauer >>> >>> GTalk: neubauer.peter >>> Skype peter.neubauer >>> Phone +46 704 106975 >>> LinkedIn http://www.linkedin.com/in/neubauer >>> Twitter http://twitter.com/peterneubauer >>> >>> http://www.neo4j.org - Your high performance graph database. >>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>> >>> >>> >>> On Thu, Sep 22, 2011 at 1:52 PM, Romiko Derbynew >>> <romiko.derby...@readify.net> wrote: >>>> Hi, >>>> >>>> It's the database logging, I'm finding the transaction logs are causing >>>> the rest API to become unresponsive when it flushes to disk. Basically >>>> just want to turn it off or set the tlog size by configuration and not via >>>> code sample as it is demonstrated in the performance guide. >>>> >>>> The current work around without this setting is a circuit breaker >>>> pattern. But I am very concerned as the DB flush should not slow >>>> down further rest calls :( >>>> >>>> Sent from my iPhone >>>> >>>> On 22/09/2011, at 6:36 PM, "Peter Neubauer" >>>> <peter.neuba...@neotechnology.com> wrote: >>>> >>>>> Romiko, >>>>> which log files are you referring to? For the database transaction >>>>> log rotation, look at the neo4j.properties file to tune things, see >>>>> http://docs.neo4j.org/chunked/snapshot/server-configuration.html >>>>> >>>>> For the server, there is logging.properties to tweak stuff. Note >>>>> that these settings require a restart to take effect. >>>>> >>>>> HTH >>>>> >>>>> Cheers, >>>>> >>>>> /peter neubauer >>>>> >>>>> GTalk: neubauer.peter >>>>> Skype peter.neubauer >>>>> Phone +46 704 106975 >>>>> LinkedIn http://www.linkedin.com/in/neubauer >>>>> Twitter http://twitter.com/peterneubauer >>>>> >>>>> http://www.neo4j.org - Your high performance graph database. >>>>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>>>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>>>> >>>>> >>>>> >>>>> On Thu, Sep 22, 2011 at 1:38 AM, Romiko Derbynew >>>>> <romiko.derby...@readify.net> wrote: >>>>>> Hi, >>>>>> >>>>>> I am using Neo4j on a Windows Server, is there any way I can configure >>>>>> log rotation via the configuration files? I would like to turn it off or >>>>>> try a different log rotation size. Since I am on windows, I am not sure >>>>>> how I would do it at runtime. >>>>>> >>>>>> Thanks! >>>>>> _______________________________________________ >>>>>> Neo4j mailing list >>>>>> User@lists.neo4j.org >>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>> >>>>> _______________________________________________ >>>>> Neo4j mailing list >>>>> User@lists.neo4j.org >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> User@lists.neo4j.org >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
