Hi Johan, Thanks for following up on this discussion.
I have new findings to share with you since GAE now works a bit differently with all these recent changes. Before I do that I will share the changes I made to our app which has helped us mitigate some performance issues. 1. We moved all our cron jobs to the Backends with 1 B1 RI. Amazingly it is able to handle all the load single handed compared to 6 instances (3 RI + 3 DI) in the regular app. We have our eye brows raised! But we are playing it with caution by setting <max-concurrent- requests> to 1. Will gradually raise the limits until it starts complaining. 2. We set Max Idle Instances to 6 and Min Pending Latency to 15s. I recommend keeping Max Idle Instances to at least 4. Because if you keep it at 3 GAE kills and creates DI "very" aggressively ... in some cases 10-20 seconds. 3. In our regular app we are running dummy cron jobs to keep 3 DI up at all times. Again, amazingly with many concurrent users they are able to handle ALL the load. On the contrary RI are almost untouched. Here's the latest statistics: QPS* Latency* Requests Errors Age Memory Availability 0.000 0.0 ms 25 0 12:43:22 82.3 MBytes Resident Icon Resident 0.000 0.0 ms 10 0 12:44:45 80.6 MBytes Resident Icon Resident 0.000 0.0 ms 5 0 12:44:46 81.9 MBytes Resident Icon Resident 0.350 19.1 ms 11762 0 9:45:11 118.5 MBytes Dynamic Icon Dynamic 0.317 17.5 ms 3890 0 3:11:04 97.7 MBytes Dynamic Icon Dynamic 0.333 67.5 ms 15604 0 12:43:41 104.8 MBytes Dynamic Icon Dynamic But this is just a sample set of users. I don't know yet if we can scale on this principle but it seems to work on light load. But this is definitely short term fix since this is going to be expensive under new billing. After spending fairly good amount of time collecting these statistics I can boil down to a few things that I want to share, that can help us tune our app to GAE. A. Know the current usage of an instance (RI or DI) in terms of current CPU utilization in percentage. This will help us correlate CPU utilization shown in each request in the logs and help us tune our queries. Right now there is no way to determine what effect the tuning had on an individual instance. B. Know the number of current threads running within an instance (maybe in a span of 15 seconds or so). This will help us identify ratio of thread handling capacity vs CPU consumed (point A). Higher the number means better throughput. Lower the number means we need to closely examine our requests. C. Able to override the 30 second deadline for the warm up requests so that we have enough time to load our frameworks. D. Able to specify how long an instance can live before reclaimed by GAE. Something similar to session timeouts. This will help avoid unnecessary warm up requests and will improve overall performance. E. Ability to specify rules within GAE console that will adjust the Max Idle Instances during different times of a day and week. F. Able to specify the size of DI (something similar to Backends B1, B2 ... so on) instead of standard DI. Should be able to configure along with each rule mentioned in point E. I understand that C, D, E, F may involve some additional costs to us but these short / long term enhancements will hugely help us run our business applications on GAE. The idea is to give us more time and capacity with the DI while keeping a control on the cost. Let me know if I missed something that is already there or I am going off tangent here. On Jul 25, 9:18 am, Johan Euphrosine <pro...@google.com> wrote: > Hi Francois, > > I think the help text of the 'Idle Instances' settings is pretty > self-explanatory: > """ > You will not be charged for instances over the specified maximum. > """ > > I can investigate on why these instances are created even thought you > setup a high min pending latency, and a a Max idle instances > corresponding to the number of Always On instance. > > Feel free to open a Production issue, with your application id if this > is affection your operation. > > Thanks in advance. > > > > > > > > > > On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel <f.masu...@gmail.com> wrote: > > Something strange : > > GAE keeps starting new dynamic instances though I have set the Min Pending > > Latency to 15s via the Application Settings page. > > My requests are served in a few ms on average and I have > > multithreading enabled. > > These instances are immediately destroyed as I have set the Max Idle > > Instances to 3 (corresponding to my always on instances). > > > But still lots of warmup requests in my logs showing these dynamic instances > > start every few minutes. > > Will I have to pay for all these short lived dynamic instances ? > > My app ID is : vncts1 > > Thanx for your help. > > Francois > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web visit > >https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Johan Euphrosine (proppy) > Developer Programs Engineer > Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
