[google-appengine] Re: Back-ends instances are dying and not restoring
Seeing the same behavior. Found the url to the issue: http://code.google.com/p/googleappengine/issues/detail?id=5465 On Aug 2, 8:06 am, Kirill Zinoviev wrote: > Done - issue posted. Thanks for advice. > If will be resolved then will post results here. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
Hi Johan, Thanks for following up on this discussion. I have new findings to share with you since GAE now works a bit differently with all these recent changes. Before I do that I will share the changes I made to our app which has helped us mitigate some performance issues. 1. We moved all our cron jobs to the Backends with 1 B1 RI. Amazingly it is able to handle all the load single handed compared to 6 instances (3 RI + 3 DI) in the regular app. We have our eye brows raised! But we are playing it with caution by setting to 1. Will gradually raise the limits until it starts complaining. 2. We set Max Idle Instances to 6 and Min Pending Latency to 15s. I recommend keeping Max Idle Instances to at least 4. Because if you keep it at 3 GAE kills and creates DI "very" aggressively ... in some cases 10-20 seconds. 3. In our regular app we are running dummy cron jobs to keep 3 DI up at all times. Again, amazingly with many concurrent users they are able to handle ALL the load. On the contrary RI are almost untouched. Here's the latest statistics: QPS*Latency*RequestsErrors Age Memory Availability 0.000 0.0 ms 25 0 12:43:22 82.3 MBytes Resident Icon Resident 0.000 0.0 ms 10 0 12:44:45 80.6 MBytes Resident Icon Resident 0.000 0.0 ms 5 0 12:44:46 81.9 MBytes Resident Icon Resident 0.350 19.1 ms 11762 0 9:45:11 118.5 MBytes Dynamic Icon Dynamic 0.317 17.5 ms 38900 3:11:04 97.7 MBytes Dynamic Icon Dynamic 0.333 67.5 ms 15604 0 12:43:41104.8 MBytes Dynamic Icon Dynamic But this is just a sample set of users. I don't know yet if we can scale on this principle but it seems to work on light load. But this is definitely short term fix since this is going to be expensive under new billing. After spending fairly good amount of time collecting these statistics I can boil down to a few things that I want to share, that can help us tune our app to GAE. A. Know the current usage of an instance (RI or DI) in terms of current CPU utilization in percentage. This will help us correlate CPU utilization shown in each request in the logs and help us tune our queries. Right now there is no way to determine what effect the tuning had on an individual instance. B. Know the number of current threads running within an instance (maybe in a span of 15 seconds or so). This will help us identify ratio of thread handling capacity vs CPU consumed (point A). Higher the number means better throughput. Lower the number means we need to closely examine our requests. C. Able to override the 30 second deadline for the warm up requests so that we have enough time to load our frameworks. D. Able to specify how long an instance can live before reclaimed by GAE. Something similar to session timeouts. This will help avoid unnecessary warm up requests and will improve overall performance. E. Ability to specify rules within GAE console that will adjust the Max Idle Instances during different times of a day and week. F. Able to specify the size of DI (something similar to Backends B1, B2 ... so on) instead of standard DI. Should be able to configure along with each rule mentioned in point E. I understand that C, D, E, F may involve some additional costs to us but these short / long term enhancements will hugely help us run our business applications on GAE. The idea is to give us more time and capacity with the DI while keeping a control on the cost. Let me know if I missed something that is already there or I am going off tangent here. On Jul 25, 9:18 am, Johan Euphrosine wrote: > Hi Francois, > > I think the help text of the 'Idle Instances' settings is pretty > self-explanatory: > """ > You will not be charged for instances over the specified maximum. > """ > > I can investigate on why these instances are created even thought you > setup a high min pending latency, and a a Max idle instances > corresponding to the number of Always On instance. > > Feel free to open a Production issue, with your application id if this > is affection your operation. > > Thanks in advance. > > > > > > > > > > On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel wrote: > > Something strange : > > GAE keeps starting new dynamic instances though I have set the Min Pending > > Latency to 15s via the Application Settings page. > > My requests are served in a few ms on average and I have > > multithreading enabled. > > These instances are immediately destroyed as I have set the Max Idle > > Instances to 3 (corresponding to my always on instances). > > > But still lots of warmup requests in my logs showing these dynamic instances > > start every few minutes. > > Will I have to pay for all these short lived dynamic instances ? > > My app ID is : vncts1 > > Thanx for your help. > > Francois >
[google-appengine] Re: GAE starting unnecessary instances
Hi Johan, Thanks for the explanation. I have couple of questions on that. 1. "1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests" While this may be true when my Always On instances were "busy" running some stuff but what about when 2 Always On instances show only "1" request served which is the Warmup request itself. Does this mean Warmup requests are considered as traffic? If that is the case then Always On instances seem rather useless since they will never ever get called in this scenario. 2. As Tom mentioned, what qualifies "busy". When threadsafe option was implemented in GAE these 3 Always On instances were able to do most of the heavy lifting with occasional spinning of dynamic instances. Nothing has changed on our side that should alter this behavior. With all these changes happening within GAE I am trying to figure out what changed and what we can do to contain this burst of traffic within 3 (or more ) Always On instances with less frequent spinning of Dynamic instances. 3. "- 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. " Again what constitutes "busy" as I do not see any request being served by Always On instances 2 and 3 in last 1 hour. Note that number of requests served by Always On 2/3 are unchanged since they were created ... Here's my reading in this scenario: a. It kills Dynamic Instance 1 within 2 minutes of serving a request b. When traffic comes in it looks only for Dynamic Instances if they are busy and completely ignores Always On instances at this point c. It recreates Dynamic Instance 1 In other words, what rule is applied in this case? Also I fail to understand rule 4 as both Rob and Luca mentioned. That completely undermines having Always On instances under threadsafe mode. 4. I like Rob's suggestion of better load balancing techniques but again with a caveat that an instance needs to be able to serve multiple threads before reaching a set capacity (80% or so) 5. Luca's suggestion also makes sense but again with the same caveat ... it should be able to process multiple threads before queuing 6. I looked at the new sliders in the Admin console and with those the situation is even worse. I set the Max Idle Instances to 3 (that's the minimum I could choose) and Min Pending Latency to 15 secs ... Guess what our CPU usage has gone up to 15 in 12 hrs because of constant creation and killing of 3 dynamic instances. Bare minimum traffic and few light weight crons. But the good side is now I see requests coming in on the 3 Always On instances. Is that enough load they are serving ... I don't know yet but something to observe. Two things I suggest would be really helpful for us: A. The overall key here is to know the thread handling capacity of an instance. Better yet if it can be configured similar to Backends but dynamic in nature (and of course Backends pricing is outrageous ... but that's another topic) B. Able to add more Always On instances but again with a dependency explained in point A. Hope it makes sense. Thanks, galoch On Jul 22, 7:57 am, Johan Euphrosine wrote: > HI Galoch, > > Thanks for the followup, > > I think you are experiencing a combinaison fo the two following rules > I was pointing to in my previous email: > (> reads as has priority for handling the incoming request) > 2/ Spawning a new Dynamic instance > Busy Always On instance > 4/ Idle Dynamic instance > Idle Always On instance > > Applied to your example it could means that: > Resident Instance 1: Requests: 49 Age: 1Hr > Resident Instance 2: Requests: 6 Age: 1Hr > Resident Instance 3: Requests: 2 Age: 1Hr > Dynamic Instance 1: Requests: 7 Age: 2min > Dynamic Instance 2: Requests: 291 Age: 1Hr > Dynamic Instance 3: Requests: 322 Age: 1Hr > > - 1 Hours ago while all your Always On instance were busy and you had > a burst of incoming requests and the scheduler spawned new Dynamic > instances as per rule 2/ highlighted above. > - After the burst and back to normal traffic the new Dynamic Instances > were handing incoming requests in priority as per rule 4/ highlighted > above. > - 2 Minutes ago all your instances Always On + Dynamic were busy again > and the scheduler spawned a new Dynamic instance that handle 7 > incoming requests. > > Hope that make more sense for you and Francois, but as I said earlier > we are open to suggestion and I will make sure someone working on the > scheduler team monitor this thread for your input. > > > > > > > > > > On Fri, Jul 22, 2011 at 9:09 AM, Galoch wrote: > > @Johan, > > The issue is not about Always On instance being busy. Its actually the >
[google-appengine] Re: GAE starting unnecessary instances
@Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under "no load" with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only "1" request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine wrote: > After speaking with Engs, I think I can explain what is going on: > > Here are the current scheduling rules: (> reads as has priority for > handling the incoming request) > > 1/ Idle Always On instance > Spawning a new Dynamic instance > 2/ Spawning a new Dynamic instance > Busy Always On instance > 3/ Idle Dynamic instance > Busy Always On instance > 4/ Idle Dynamic instance > Idle Always On instance > > I will give you an example to illustrate the behavior you all noticed, > that is Dynamic instance handling request while Always On is idle. > > (Always On instance started) > - Incoming request > - Always On instance handle the request > - another Incoming request > (Always On instance busy) > - A new Dynamic instance is spawned > (Dynamic instance idle, Always on instance busy) > - Dynamic instance handle the request > - another Incoming request > (Dynamic instance idle, Always on instance idle) > - Dynamic instance handle the request > - No request for more than idle-dynamic-instance-timeout > - Dynamic instance shut down > - another Incoming request > (Always On instance idle) > - Always On instance handle the request > > Hope it makes thing clearer. > > As part of the new billing model you will have a scheduler knob called > 'max-idle-instances' that you can use if extra idling dynamic > instances are undesired. > > The good news is that we are open to suggestion, if you think this > behavior is the wrong default, feel free to comment on that thread and > I will follow up your suggestion to the Engineering team. > > > > > > > > > > On Wed, Jul 20, 2011 at 12:18 AM, Galoch wrote: > > Same here. Seems like GAE is totally ignoring Always On instances. > > I also noticed that even with no user hitting our app and a single > > cron job that runs every 5 minutes it is still spinning instances > > every 3 minutes and then killing them in 2 minutes. > > > This has been happening since after the upgrade on 14th July. During > > peak load this really gets nasty and brings down the performance. > > > This is the feed
[google-appengine] Re: GAE starting unnecessary instances
Same here. Seems like GAE is totally ignoring Always On instances. I also noticed that even with no user hitting our app and a single cron job that runs every 5 minutes it is still spinning instances every 3 minutes and then killing them in 2 minutes. This has been happening since after the upgrade on 14th July. During peak load this really gets nasty and brings down the performance. This is the feedback I got yesterday from one of our customers since it takes time to spin an instance (and yes we use Spring): "1) I found the GUI to be very laggy" Can someone from Google please respond? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Always On - Amount of Loading Requests the same
Did you find any solution? I am seeing the exact behavior ... but instead of 3 Always On instances I only have 2 Always On instances and 1 dynamic instance. Interestingly, I am seeing my tasks executing on dynamic instance that results in a loading request. I use Spring framework and it is very expensive on loading requests ... so some of them end up in Hard deadline exceeded error. Right now there is very little traffic on my site and I am wondering why there is Dynamic instance at all? I am desperately seeking a solution Thanks galoch On Jan 17, 8:58 pm, brianl wrote: > Doesn't that then beg the question of how many requests one 'Always > On' instance could handle and what's the typical performance of an > instance in terms of CPU, network requests, etc? > > With 8 instances I would guess there's some distributed caching > overhead that may be unwanted or undesirable if the app doesn't have > lots of traffic. > > -brian > > On Jan 17, 8:42 pm, Robert Kluin wrote: > > > > > > > > > Out of curiosity, why? Generally speaking, the more instances your > > app has the more likely it is that a new request will be serviced > > quickly. You are not charged for the additional instances, unless > > they are doing something (in which case you probably needed them > > anyway). > > > Robert > > > On Mon, Jan 17, 2011 at 22:35, brianl wrote: > > > Just checked my Instances allocations... Seeing the 3 'AlwaysOn' > > > instances and 5 Dynamic instances. The 5 Dynamic instances seems > > > unusually high. So guess I have 8 instances total. There's very > > > little traffic to the app to cause so many instances to be running. > > > I'd like to be just running the 3 'AlwaysOn' instances and no Dynamic > > > instances, is this possible? If so where's the setting to have NO > > > Dynamic instances running when 'AlwaysOn' is enabled? > > > > On Jan 17, 1:41 pm, "Brandon Wirtz" wrote: > > >>Alwayson helps if you only need 3 instances to keep things running. If > > >> your app requires 4 instances be running all the time you won't see any > > >> benefit. If on the other hand you serve one request an hour, you'd see a > > >> lot of benefit on that one request. > > > >> Without knowing what you are doing, or why you are doing it we can't help > > >> you diagnose if what you are doing is sane and rational, or if it is > > >> something in your design that is causing the issue. > > > >> From: google-appengine@googlegroups.com > > >> [mailto:google-appengine@googlegroups.com] On Behalf Of brianl > > >> Sent: Monday, January 17, 2011 11:53 AM > > >> To: google-appengine@googlegroups.com > > >> Subject: [google-appengine]AlwaysOn - Amount of Loading Requests the same > > > >> According to the docs the 'AlwaysOn' feature mitigates the impact of > > >> loading requests. I've been running w/ the 'AlwaysOn' feature for > > >> several > > >> days. I thought I'd see far fewer loading requests, but it appears to be > > >> the same as before. There are still cases where I'll see loading > > >> requests > > >> just a few minutes apart. Most times I see loading requests occur about > > >> 50~60 minutes apart. So far I'm not seeing the benefit of the 'AlwaysOn' > > >> feature. It appears to be a waste of money. > > > >> -- > > >> You received this message because you are subscribed to the Google Groups > > >> "Google App Engine" group. > > >> To post to this group, send email to google-appengine@googlegroups.com. > > >> To unsubscribe from this group, send email to > > >> google-appengine+unsubscr...@googlegroups.com. > > >> For more options, visit this group > > >> athttp://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appengine@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com. > > > For more options, visit this group > > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Issue with precompilation-enabled java sdk 1.3.5
Interesting. Now I am able to deploy my app with precompilation- enabled set to true without making any change to app or environment. Thanks G On Jul 19, 11:41 pm, Galoch wrote: > Hi, > > I looked at the release notes of 1.3.1 where it says "Java > Precompilation is now on by default" where as the current > documentation at link [http://code.google.com/appengine/kb/java.html] > says it will be supported in future versions. > > I believe, former is true. Reason being, I am unable to deploy > application with default settings in my appengine-web.xml since > deployment fails with 500 internal server error. Has anyone else > noticed it. I have seen this issue pop up after I started using task > queues. > After explicitly turning off precompilation of JSPs, I am able to > deploy but I am not sure what would be the impact on overall app > performance. > > Thanks > G -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Issue with precompilation-enabled java sdk 1.3.5
Hi, I looked at the release notes of 1.3.1 where it says "Java Precompilation is now on by default" where as the current documentation at link [http://code.google.com/appengine/kb/java.html] says it will be supported in future versions. I believe, former is true. Reason being, I am unable to deploy application with default settings in my appengine-web.xml since deployment fails with 500 internal server error. Has anyone else noticed it. I have seen this issue pop up after I started using task queues. After explicitly turning off precompilation of JSPs, I am able to deploy but I am not sure what would be the impact on overall app performance. Thanks G -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
