Re: [google-appengine] Re: GAE starting unnecessary instances
The gist of the discussion is: - Idle Dynamic Instances get traffic in priority over Idle Always On Instances. - People have shown in the thread that 'Max Idle Instance' knob could be used to workaround this behavior by preventing too many Dynamic Instances from spawning. - This is not really worth worrying before the new billing model, since the current behavior is temporary until Always On Instances get superseeded by 'Min Idle Instance' knob. A few suggestions coming out of this thread that I forwarded the scheduler engineering team: 1/ Show the current CPU% usage of an instance. 2/ Show the current number of threads of an instance. 3/ Allow to override the 30 second deadline for the warm up requests (for loading frameworks). 4/ New idle_instance_timeout knob: how long an idle instance can live before being reclaimed. 5/ Allow to configure the capacity for Dynamic Instance (in a similar way to Backends). 6/ Add rules for changing Max Idle Instances / Instance capacity during different times of a day and week. 7/ Multiple high availability preset regarding instance spawning: ( safe + unpredicatable cost ) versus ( unsafe + predictable cost ). 8/ Allow user to specify a budget related to request serving delay so the scheduler can synthesize a policy about instance spawning. 9/ Alternative load balancing strategy: load 1 instance 50% before using the second one and so on (instead of round-robin). Hope that helps. On Wed, Jul 27, 2011 at 1:47 AM, Santiago Lema jacques.l...@gmail.com wrote: To keep things simple, is this issue worth worrying about before the new billing model becomes active ? Or will everything be so different that the stuff we see know will be different then ? My instances: https://skitch.com/smalltech/fcyfd/instances-smallte.ch Site: smallte.ch (on http://websmalltech.appspot.com/ ) On 25 juil, 10:33, Johan Euphrosine pro...@google.com wrote: What is your setting for 'Max IdleInstances' ? Feel free to open a Production issue with you appid if you want me to track this specifically. On Sat, Jul 23, 2011 at 11:01 AM, Francois Masurel f.masu...@gmail.com wrote: Geez, now even some Dynamicinstancesare not used too and stay idle for hours (check attachment). With the new pricing model this will get really really expensive. I hope there is definitely something going wrong. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Qczx6foLplUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
It seems like the service is moving in a direction in which devs will have to adjust many knobs, rather than have Google handle it automagically in the background. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
Actually the latter knobs I listed were suggestion from the community but feel free to raise a different voice. The only knobs Google provide so far are: - maximum number of idle instances - minimum pending latency See http://code.google.com/appengine/docs/adminconsole/performancesettings.html for more details. - min number of idle instances will come later to supplant Always on. And all knobs should have automatic default values that make App Engine handle those automagically in the background. On Wed, Jul 27, 2011 at 12:20 PM, pdknsk pdk...@googlemail.com wrote: It seems like the service is moving in a direction in which devs will have to adjust many knobs, rather than have Google handle it automagically in the background. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
On Mon, Jul 25, 2011 at 9:25 PM, Galoch galoch...@gmail.com wrote: Hi Johan, Thanks for following up on this discussion. I have new findings to share with you since GAE now works a bit differently with all these recent changes. Before I do that I will share the changes I made to our app which has helped us mitigate some performance issues. 1. We moved all our cron jobs to the Backends with 1 B1 RI. Amazingly it is able to handle all the load single handed compared to 6 instances (3 RI + 3 DI) in the regular app. We have our eye brows raised! But we are playing it with caution by setting max-concurrent- requests to 1. Will gradually raise the limits until it starts complaining. 2. We set Max Idle Instances to 6 and Min Pending Latency to 15s. I recommend keeping Max Idle Instances to at least 4. Because if you keep it at 3 GAE kills and creates DI very aggressively ... in some cases 10-20 seconds. Thanks for investigating those behaviors, feel free to continue to share your findings with the community. 3. In our regular app we are running dummy cron jobs to keep 3 DI up at all times. Again, amazingly with many concurrent users they are able to handle ALL the load. On the contrary RI are almost untouched. Here's the latest statistics: QPS* Latency* Requests Errors Age Memory Availability 0.000 0.0 ms 25 0 12:43:22 82.3 MBytes Resident Icon Resident 0.000 0.0 ms 10 0 12:44:45 80.6 MBytes Resident Icon Resident 0.000 0.0 ms 5 0 12:44:46 81.9 MBytes Resident Icon Resident 0.350 19.1 ms 11762 0 9:45:11 118.5 MBytes Dynamic Icon Dynamic 0.317 17.5 ms 3890 0 3:11:04 97.7 MBytes Dynamic Icon Dynamic 0.333 67.5 ms 15604 0 12:43:41 104.8 MBytes Dynamic Icon Dynamic But this is just a sample set of users. I don't know yet if we can scale on this principle but it seems to work on light load. But this is definitely short term fix since this is going to be expensive under new billing. I believe those stats illustrate the rule we discussed before: Idle DI Idle RI. Keep in mind that under the new pricing model Min Idle Instances should superseed Always On instances, as described by gregd in the billing faq. So hopefully you should not have idling DI instance wandering around, and you will be able to control this with Max Idle Instances anyway. After spending fairly good amount of time collecting these statistics I can boil down to a few things that I want to share, that can help us tune our app to GAE. A. Know the current usage of an instance (RI or DI) in terms of current CPU utilization in percentage. This will help us correlate CPU utilization shown in each request in the logs and help us tune our queries. Right now there is no way to determine what effect the tuning had on an individual instance. B. Know the number of current threads running within an instance (maybe in a span of 15 seconds or so). This will help us identify ratio of thread handling capacity vs CPU consumed (point A). Higher the number means better throughput. Lower the number means we need to closely examine our requests. C. Able to override the 30 second deadline for the warm up requests so that we have enough time to load our frameworks. D. Able to specify how long an instance can live before reclaimed by GAE. Something similar to session timeouts. This will help avoid unnecessary warm up requests and will improve overall performance. E. Ability to specify rules within GAE console that will adjust the Max Idle Instances during different times of a day and week. F. Able to specify the size of DI (something similar to Backends B1, B2 ... so on) instead of standard DI. Should be able to configure along with each rule mentioned in point E. I understand that C, D, E, F may involve some additional costs to us but these short / long term enhancements will hugely help us run our business applications on GAE. The idea is to give us more time and capacity with the DI while keeping a control on the cost. I think your suggestions would be a great addition to the admin-console, and let user have more visibility and control over their instances and thus over billing. You should definitely fill separate feature requests on the public issue tracker for them. In the meantime, I will make sure the engineering team is aware of your suggestion. Let me know if I missed something that is already there or I am going off tangent here. You're completely on the topic of this discussion :) On Jul 25, 9:18 am, Johan Euphrosine pro...@google.com wrote: Hi Francois, I think the help text of the 'Idle Instances' settings is pretty self-explanatory: You will not be charged for instances over the specified
Re: [google-appengine] Re: GAE starting unnecessary instances
On Fri, Jul 22, 2011 at 5:37 PM, Rob Coops rco...@gmail.com wrote: 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance So App engine prefers to use bored Always On instances over spawning new dynamic once that's good. If the Always On instances are busy it spawns a new Dynamic instance good. If a Dynamic instance is bored but the Always On once are busy the Dynamic instance gets the load still good. But then you loose me if the Always one instance is idle the Dynamic instance still gets the load, why? In this last case I would expect the Always on instance to get the load otherwise the Dynamic instance will keep on being busy and will not get stopped because of it. I don't know what the cost are of spawning and later destroying a Dynamic instance but I cannot imagine that this is such a huge cost that you would have to prefer using the Dynamic instances over the Always On once. I believe that this last rule should read: 4/ Idle Always On instance Idle Dynamic instance In which case the Idle Always On instances would get the load and the bored Dynamic instances would get cleaned up much faster then they are now. I suspect this is a typo though as I cannot imagine that this is really the setup but if it is I would say that is a candidate for change. :-) This is not a typo: 4/ Idle Always On instance Idle Dynamic instance Under the new pricing model, as gregd highlighted in the pricing faq Always On will be superseeded with Min Idle Instances. So we shouldn't see that weird behaviour of Always On instance getting idle, but rather a nice distribution of Dynamic Instance matching the scheduler knobs you set in admin-console. The other thing I would suggest is altering the load balancing rules for Dynamic instances from the picture painted in this email it looks like the load balancing of multiple Dynamic instances is pretty much round robin (or equal load based). If this would be changed to always try and load use one Dynamic instance till the load reaches 80% or so before using the second one and so on this would allow the despawning of excess Dynamic instances much sooner then when one uses the current setup. This does mean a slightly bigger hit in case of a serious failure of the currently preferred Dynamic instance. Hence the 80% mark for the load which is arbitrarily chosen by randomly picking a number above 50 and might need some more scientific work to ensure that in an average scenario the remaining instances will usually be able to take the load caused by the sudden death of an Dynamic instance. From what I currently see it looks like the safest option has been chosen, meaning that in all cases the service will remain active no matter what happens but this means a significant cost on the customer side. I suspect that many customers are happy with that, but I think that an equal amount of them will want to see a situation where the costs are less likely to spike while providing a similar albeit slightly less high availability solution. It might be an interesting idea to offer several flavors of high availability, ranging from the current supper safe but relatively unpredictable cost to a pretty decent with very predictable costs one. I have no idea if this is technically possible but something tells me that it should not be that hard to do. And even if it is a little harder to do my guess is that Google would be up to that task. Well that's my two cents... Thanks a lot for those suggestions, I will make sure to forward them to the engineering team. In the meantime feel free to open feature request for them if you want to the community to be able to track their progress. ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance On Fri, Jul 22, 2011 at 4:57 PM, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to
[google-appengine] Re: GAE starting unnecessary instances
To keep things simple, is this issue worth worrying about before the new billing model becomes active ? Or will everything be so different that the stuff we see know will be different then ? My instances: https://skitch.com/smalltech/fcyfd/instances-smallte.ch Site: smallte.ch (on http://websmalltech.appspot.com/ ) On 25 juil, 10:33, Johan Euphrosine pro...@google.com wrote: What is your setting for 'Max IdleInstances' ? Feel free to open a Production issue with you appid if you want me to track this specifically. On Sat, Jul 23, 2011 at 11:01 AM, Francois Masurel f.masu...@gmail.com wrote: Geez, now even some Dynamicinstancesare not used too and stay idle for hours (check attachment). With the new pricing model this will get really really expensive. I hope there is definitely something going wrong. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Qczx6foLplUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
On Fri, Jul 22, 2011 at 8:57 PM, Galoch galoch...@gmail.com wrote: Hi Johan, Thanks for the explanation. I have couple of questions on that. Thanks for showing interest in GAE internals, I'd be happy to answer those questions directly if I can, or forward them to someone who can answer them better. 1. 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests While this may be true when my Always On instances were busy running some stuff but what about when 2 Always On instances show only 1 request served which is the Warmup request itself. Does this mean Warmup requests are considered as traffic? If that is the case then Always On instances seem rather useless since they will never ever get called in this scenario. On the admin console capture you included in your previous mail, I didn't see Always On instances showing only 1 request served but rather: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Let me know if I missed something. 2. As Tom mentioned, what qualifies busy. When threadsafe option was implemented in GAE these 3 Always On instances were able to do most of the heavy lifting with occasional spinning of dynamic instances. Nothing has changed on our side that should alter this behavior. With all these changes happening within GAE I am trying to figure out what changed and what we can do to contain this burst of traffic within 3 (or more ) Always On instances with less frequent spinning of Dynamic instances. There are two scheduler knobs that could help you to affect the way Dynamic instance are spawned. Minimum Pending Latency and Max Idle Instances as described here: http://code.google.com/appengine/docs/adminconsole/performancesettings.html 3. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Again what constitutes busy as I do not see any request being served by Always On instances 2 and 3 in last 1 hour. Note that number of requests served by Always On 2/3 are unchanged since they were created ... Here's my reading in this scenario: a. It kills Dynamic Instance 1 within 2 minutes of serving a request b. When traffic comes in it looks only for Dynamic Instances if they are busy and completely ignores Always On instances at this point c. It recreates Dynamic Instance 1 In other words, what rule is applied in this case? Sorry, those were mostly specification of mine, I didn't know that the request served by Always On 2/3 were unchanged according to the information you provided. I can investigate deeper into the specific behaviour of your application, if you open a Production Issue with your application id. Also I fail to understand rule 4 as both Rob and Luca mentioned. That completely undermines having Always On instances under threadsafe mode. 4. I like Rob's suggestion of better load balancing techniques but again with a caveat that an instance needs to be able to serve multiple threads before reaching a set capacity (80% or so) 5. Luca's suggestion also makes sense but again with the same caveat ... it should be able to process multiple threads before queuing Thanks a lot for your feedback, I will make sure to forward those suggestions to the engineering team. 6. I looked at the new sliders in the Admin console and with those the situation is even worse. I set the Max Idle Instances to 3 (that's the minimum I could choose) and Min Pending Latency to 15 secs ... Guess what our CPU usage has gone up to 15 in 12 hrs because of constant creation and killing of 3 dynamic instances. Bare minimum traffic and few light weight crons. But the good side is now I see requests coming in on the 3 Always On instances. Is that enough load they are serving ... I don't know yet but something to observe. Maybe you can open a feature request for having a smaller min for 'Max Idle Instance' when Always On is activated or having Always On instances count in Max Idle Instance. Two things I suggest would be really helpful for us: A. The overall key here is to know the thread handling capacity of an instance. Better yet if it can be configured similar to Backends but dynamic in nature (and of course Backends pricing is outrageous ... but that's another topic) Are you looking for max-concurrent-requests support for Servlet ? If so I would recommend to open a Feature request. B. Able to add more Always On instances but again with a dependency explained in point A. Again, opening a feature request make sense to track this separately. On Jul 22, 7:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming
Re: [google-appengine] Re: GAE starting unnecessary instances
Hi, I will ask to the engineer team and get back to you in thread. Hope that helps. On Fri, Jul 22, 2011 at 5:33 PM, Tom Phillips tphill0...@gmail.com wrote: When are threaded instances considered busy? If it is while they are serving only a one request it would explain why Always on instances for threaded Java are now severely under- utilized. /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/
Re: [google-appengine] Re: GAE starting unnecessary instances
On Fri, Jul 22, 2011 at 7:09 PM, Luca luca.de.alf...@gmail.com wrote: Johan, it seems to me the problem is rule 4/ below. Shouldn't the rule be: Idle Always On instance Idle Dynamic instance instead? In this way, if you have a dynamic instance on, it will not handle traffic unless the always-on ones are busy. So when the traffic decreases, and the dynamic instances are no longer needed, they will be correctly shut off. I confirm the current rules is Idle Dynamic instance Always On instance While I agree your proposition make sense in some cases, it sounds also good to take advantage of the Dynamic instances that have just be spawned especially if the startup cost is high. Like Gregd highlighted in his pricing faq, Always On will became a settings to control the number of idle instances you would like to have running. Q: How will Always On work under the new model? A: When App Engine leaves preview all Paid Apps and Apps in Premier Accounts will be able to set the number of idle instances they would like to have running. Always On was designed to allow an app to always have idle instances running to save on instance start-up latency. For many Apps a single idle instance should be enough (especially when using concurrent requests). This means that for many customers, setting an App to be paid will mean a $9/month minimum spend, you can then use the 24 free IH/day to keep an instance running all the time by setting Min Idle Instances to be 1. I also think an auxiliary problem is with rule 2/ If you spawn a new instance just because the others are all busy, you may tend to have too many instances. The fact of spawning new instances should be configurable, depending on how many requests are already queued for the instances you have already active -- 0, 1, more, etc. More in general, if you allowed users to specify how much it costs to them to delay serving a request, it would not be difficult to synthesize for each app an optimal decision policy to decide whether to switch another instance on or off. This can be done using tools from dynamic optimization / control theory. I would be glad to help if the people there need guidance on this (I used to be in the research group there till a month ago). It sounds to me that the scheduler knobs are there precisely to address this: http://code.google.com/appengine/docs/adminconsole/performancesettings.html Let me know if I overlooked it. Luca On Thursday, July 21, 2011 5:56:42 PM UTC-7, Johan Euphrosine (Google) wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/X8gH6jjIK0MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
I think Ggegd covered this in his pricing FAQ: Q: How will Always On work under the new model? A: When App Engine leaves preview all Paid Apps and Apps in Premier Accounts will be able to set the number of idle instances they would like to have running. Always On was designed to allow an app to always have idle instances running to save on instance start-up latency. For many Apps a single idle instance should be enough (especially when using concurrent requests). This means that for many customers, setting an App to be paid will mean a $9/month minimum spend, you can then use the 24 free IH/day to keep an instance running all the time by setting Min Idle Instances to be 1. Let me know if you need more information. On Fri, Jul 22, 2011 at 9:46 PM, Tom Phillips tphill0...@gmail.com wrote: The current behavior makes me suspect we are being prepared for Always- on being replaced completely by the new scheduler knobs. Being able to turn up the number of idle instances does make always-on somewhat redundant, as long as the idle instances stick around for a while. Also, if always-on instances WERE being properly utilized right now, it would require artificial load to ascertain the effects of the new scheduler on dynamic instances (assuming the three AO instances were sufficient for your app previously). Many devs wouldn't learn about the new scheduler attributes until after their app (suddenly) becomes popular - not the best time for surprises. And Google wouldn't get as much feedback on the features/behavior of the new scheduler. Is Always-on going to be kept under the new model? /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive.
Re: [google-appengine] Re: GAE starting unnecessary instances
What is your setting for 'Max Idle Instances' ? Feel free to open a Production issue with you appid if you want me to track this specifically. On Sat, Jul 23, 2011 at 11:01 AM, Francois Masurel f.masu...@gmail.com wrote: Geez, now even some Dynamic instances are not used too and stay idle for hours (check attachment). With the new pricing model this will get really really expensive. I hope there is definitely something going wrong. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Qczx6foLplUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
Hi Johan, Thanx for your answer. Setting MaxIdleInstances to 3 did force the resident instances to be used instead of the dynamic ones. As we will soon pay for active instances, I would haved like to have the 3 resident ones being used instead of extra dynamic ones being started. Thanx again for your help. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2E_xthK5Xo4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
Something strange : GAE keeps starting new dynamic instances though I have set the Min Pending Latency to 15s via the Application Settings page. My requests are served in a few ms on average and I have multithreading enabled. These instances are immediately destroyed as I have set the Max Idle Instances to 3 (corresponding to my always on instances). But still lots of warmup requests in my logs showing these dynamic instances start every few minutes. Will I have to pay for all these short lived dynamic instances ? My app ID is : vncts1 Thanx for your help. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: GAE starting unnecessary instances
Hi Francois, I think the help text of the 'Idle Instances' settings is pretty self-explanatory: You will not be charged for instances over the specified maximum. I can investigate on why these instances are created even thought you setup a high min pending latency, and a a Max idle instances corresponding to the number of Always On instance. Feel free to open a Production issue, with your application id if this is affection your operation. Thanks in advance. On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel f.masu...@gmail.com wrote: Something strange : GAE keeps starting new dynamic instances though I have set the Min Pending Latency to 15s via the Application Settings page. My requests are served in a few ms on average and I have multithreading enabled. These instances are immediately destroyed as I have set the Max Idle Instances to 3 (corresponding to my always on instances). But still lots of warmup requests in my logs showing these dynamic instances start every few minutes. Will I have to pay for all these short lived dynamic instances ? My app ID is : vncts1 Thanx for your help. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
Hi Johan, Thanks for following up on this discussion. I have new findings to share with you since GAE now works a bit differently with all these recent changes. Before I do that I will share the changes I made to our app which has helped us mitigate some performance issues. 1. We moved all our cron jobs to the Backends with 1 B1 RI. Amazingly it is able to handle all the load single handed compared to 6 instances (3 RI + 3 DI) in the regular app. We have our eye brows raised! But we are playing it with caution by setting max-concurrent- requests to 1. Will gradually raise the limits until it starts complaining. 2. We set Max Idle Instances to 6 and Min Pending Latency to 15s. I recommend keeping Max Idle Instances to at least 4. Because if you keep it at 3 GAE kills and creates DI very aggressively ... in some cases 10-20 seconds. 3. In our regular app we are running dummy cron jobs to keep 3 DI up at all times. Again, amazingly with many concurrent users they are able to handle ALL the load. On the contrary RI are almost untouched. Here's the latest statistics: QPS*Latency*RequestsErrors Age Memory Availability 0.000 0.0 ms 25 0 12:43:22 82.3 MBytes Resident Icon Resident 0.000 0.0 ms 10 0 12:44:45 80.6 MBytes Resident Icon Resident 0.000 0.0 ms 5 0 12:44:46 81.9 MBytes Resident Icon Resident 0.350 19.1 ms 11762 0 9:45:11 118.5 MBytes Dynamic Icon Dynamic 0.317 17.5 ms 38900 3:11:04 97.7 MBytes Dynamic Icon Dynamic 0.333 67.5 ms 15604 0 12:43:41104.8 MBytes Dynamic Icon Dynamic But this is just a sample set of users. I don't know yet if we can scale on this principle but it seems to work on light load. But this is definitely short term fix since this is going to be expensive under new billing. After spending fairly good amount of time collecting these statistics I can boil down to a few things that I want to share, that can help us tune our app to GAE. A. Know the current usage of an instance (RI or DI) in terms of current CPU utilization in percentage. This will help us correlate CPU utilization shown in each request in the logs and help us tune our queries. Right now there is no way to determine what effect the tuning had on an individual instance. B. Know the number of current threads running within an instance (maybe in a span of 15 seconds or so). This will help us identify ratio of thread handling capacity vs CPU consumed (point A). Higher the number means better throughput. Lower the number means we need to closely examine our requests. C. Able to override the 30 second deadline for the warm up requests so that we have enough time to load our frameworks. D. Able to specify how long an instance can live before reclaimed by GAE. Something similar to session timeouts. This will help avoid unnecessary warm up requests and will improve overall performance. E. Ability to specify rules within GAE console that will adjust the Max Idle Instances during different times of a day and week. F. Able to specify the size of DI (something similar to Backends B1, B2 ... so on) instead of standard DI. Should be able to configure along with each rule mentioned in point E. I understand that C, D, E, F may involve some additional costs to us but these short / long term enhancements will hugely help us run our business applications on GAE. The idea is to give us more time and capacity with the DI while keeping a control on the cost. Let me know if I missed something that is already there or I am going off tangent here. On Jul 25, 9:18 am, Johan Euphrosine pro...@google.com wrote: Hi Francois, I think the help text of the 'Idle Instances' settings is pretty self-explanatory: You will not be charged for instances over the specified maximum. I can investigate on why these instances are created even thought you setup a high min pending latency, and a a Max idle instances corresponding to the number of Always On instance. Feel free to open a Production issue, with your application id if this is affection your operation. Thanks in advance. On Mon, Jul 25, 2011 at 4:58 PM, Francois Masurel f.masu...@gmail.com wrote: Something strange : GAE keeps starting new dynamic instances though I have set the Min Pending Latency to 15s via the Application Settings page. My requests are served in a few ms on average and I have multithreading enabled. These instances are immediately destroyed as I have set the Max Idle Instances to 3 (corresponding to my always on instances). But still lots of warmup requests in my logs showing these dynamic instances start every few minutes. Will I have to pay for all these short lived dynamic instances ? My app ID is : vncts1 Thanx for your help. Francois
[google-appengine] Re: GAE starting unnecessary instances
Geez, now even some Dynamic instances are not used too and stay idle for hours (check attachment). With the new pricing model this will get really really expensive. I hope there is definitely something going wrong. Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Qczx6foLplUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. attachment: VinoCities-Instances_2.jpg
[google-appengine] Re: GAE starting unnecessary instances
@Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request while Always On is idle. (Always On instance started) - Incoming request - Always On instance handle the request - another Incoming request (Always On instance busy) - A new Dynamic instance is spawned (Dynamic instance idle, Always on instance busy) - Dynamic instance handle the request - another Incoming request (Dynamic instance idle, Always on instance idle) - Dynamic instance handle the request - No request for more than idle-dynamic-instance-timeout - Dynamic instance shut down - another Incoming request (Always On instance idle) - Always On instance handle the request Hope it makes thing clearer. As part of the new billing model you will have a scheduler knob called 'max-idle-instances' that you can use if extra idling dynamic instances are undesired. The good news is that we are open to suggestion, if you think this behavior is the wrong default, feel free to comment on that thread and I will follow up your suggestion to the Engineering team. On Wed, Jul 20, 2011 at 12:18 AM, Galoch galoch...@gmail.com wrote: Same here. Seems like GAE is totally ignoring Always On instances. I also noticed that even with no user hitting our app and a single cron job that runs every 5 minutes it is still spinning instances every 3 minutes and then killing them in 2 minutes. This has been happening since after the upgrade on 14th July. During peak load this really gets nasty and brings down the performance. This is the feedback I got yesterday from one of our customers since it takes time to spin an instance (and yes we use Spring): 1) I found the GUI to be very laggy Can someone from Google please respond? -- You received this message because you are subscribed to the Google Groups Google App Engine
[google-appengine] Re: GAE starting unnecessary instances
Still very close from Galoch situation, mine is a bit different. My Dynamic instances dont seem to be recycled but still my Resident instances dont seem to be used at all (see attachment). Why ? Francois -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_r0g9FOzxdwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. attachment: Instances-VinoCities.jpg
Re: [google-appengine] Re: GAE starting unnecessary instances
HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request while Always On is idle. (Always On instance started) - Incoming request - Always On instance handle the request - another Incoming request (Always On instance busy) - A new Dynamic instance is spawned (Dynamic instance idle, Always on instance busy) - Dynamic instance handle the request - another Incoming request (Dynamic instance
[google-appengine] Re: GAE starting unnecessary instances
When are threaded instances considered busy? If it is while they are serving only a one request it would explain why Always on instances for threaded Java are now severely under- utilized. /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request
Re: [google-appengine] Re: GAE starting unnecessary instances
1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance So App engine prefers to use bored Always On instances over spawning new dynamic once that's good. If the Always On instances are busy it spawns a new Dynamic instance good. If a Dynamic instance is bored but the Always On once are busy the Dynamic instance gets the load still good. But then you loose me if the Always one instance is idle the Dynamic instance still gets the load, why? In this last case I would expect the Always on instance to get the load otherwise the Dynamic instance will keep on being busy and will not get stopped because of it. I don't know what the cost are of spawning and later destroying a Dynamic instance but I cannot imagine that this is such a huge cost that you would have to prefer using the Dynamic instances over the Always On once. I believe that this last rule should read: 4/ Idle Always On instance Idle Dynamic instance In which case the Idle Always On instances would get the load and the bored Dynamic instances would get cleaned up much faster then they are now. I suspect this is a typo though as I cannot imagine that this is really the setup but if it is I would say that is a candidate for change. :-) The other thing I would suggest is altering the load balancing rules for Dynamic instances from the picture painted in this email it looks like the load balancing of multiple Dynamic instances is pretty much round robin (or equal load based). If this would be changed to always try and load use one Dynamic instance till the load reaches 80% or so before using the second one and so on this would allow the despawning of excess Dynamic instances much sooner then when one uses the current setup. This does mean a slightly bigger hit in case of a serious failure of the currently preferred Dynamic instance. Hence the 80% mark for the load which is arbitrarily chosen by randomly picking a number above 50 and might need some more scientific work to ensure that in an average scenario the remaining instances will usually be able to take the load caused by the sudden death of an Dynamic instance. From what I currently see it looks like the safest option has been chosen, meaning that in all cases the service will remain active no matter what happens but this means a significant cost on the customer side. I suspect that many customers are happy with that, but I think that an equal amount of them will want to see a situation where the costs are less likely to spike while providing a similar albeit slightly less high availability solution. It might be an interesting idea to offer several flavors of high availability, ranging from the current supper safe but relatively unpredictable cost to a pretty decent with very predictable costs one. I have no idea if this is technically possible but something tells me that it should not be that hard to do. And even if it is a little harder to do my guess is that Google would be up to that task. Well that's my two cents... Regards, Rob ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance On Fri, Jul 22, 2011 at 4:57 PM, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that
Re: [google-appengine] Re: GAE starting unnecessary instances
Johan, it seems to me the problem is rule 4/ below. Shouldn't the rule be: Idle Always On instance Idle Dynamic instance instead? In this way, if you have a dynamic instance on, it will not handle traffic unless the always-on ones are busy. So when the traffic decreases, and the dynamic instances are no longer needed, they will be correctly shut off. I also think an auxiliary problem is with rule 2/ If you spawn a new instance just because the others are all busy, you may tend to have too many instances. The fact of spawning new instances should be configurable, depending on how many requests are already queued for the instances you have already active -- 0, 1, more, etc. More in general, if you allowed users to specify how much it costs to them to delay serving a request, it would not be difficult to synthesize for each app an optimal decision policy to decide whether to switch another instance on or off. This can be done using tools from dynamic optimization / control theory. I would be glad to help if the people there need guidance on this (I used to be in the research group there till a month ago). Luca On Thursday, July 21, 2011 5:56:42 PM UTC-7, Johan Euphrosine (Google) wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/X8gH6jjIK0MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
Hi Johan, Thanks for the explanation. I have couple of questions on that. 1. 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests While this may be true when my Always On instances were busy running some stuff but what about when 2 Always On instances show only 1 request served which is the Warmup request itself. Does this mean Warmup requests are considered as traffic? If that is the case then Always On instances seem rather useless since they will never ever get called in this scenario. 2. As Tom mentioned, what qualifies busy. When threadsafe option was implemented in GAE these 3 Always On instances were able to do most of the heavy lifting with occasional spinning of dynamic instances. Nothing has changed on our side that should alter this behavior. With all these changes happening within GAE I am trying to figure out what changed and what we can do to contain this burst of traffic within 3 (or more ) Always On instances with less frequent spinning of Dynamic instances. 3. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Again what constitutes busy as I do not see any request being served by Always On instances 2 and 3 in last 1 hour. Note that number of requests served by Always On 2/3 are unchanged since they were created ... Here's my reading in this scenario: a. It kills Dynamic Instance 1 within 2 minutes of serving a request b. When traffic comes in it looks only for Dynamic Instances if they are busy and completely ignores Always On instances at this point c. It recreates Dynamic Instance 1 In other words, what rule is applied in this case? Also I fail to understand rule 4 as both Rob and Luca mentioned. That completely undermines having Always On instances under threadsafe mode. 4. I like Rob's suggestion of better load balancing techniques but again with a caveat that an instance needs to be able to serve multiple threads before reaching a set capacity (80% or so) 5. Luca's suggestion also makes sense but again with the same caveat ... it should be able to process multiple threads before queuing 6. I looked at the new sliders in the Admin console and with those the situation is even worse. I set the Max Idle Instances to 3 (that's the minimum I could choose) and Min Pending Latency to 15 secs ... Guess what our CPU usage has gone up to 15 in 12 hrs because of constant creation and killing of 3 dynamic instances. Bare minimum traffic and few light weight crons. But the good side is now I see requests coming in on the 3 Always On instances. Is that enough load they are serving ... I don't know yet but something to observe. Two things I suggest would be really helpful for us: A. The overall key here is to know the thread handling capacity of an instance. Better yet if it can be configured similar to Backends but dynamic in nature (and of course Backends pricing is outrageous ... but that's another topic) B. Able to add more Always On instances but again with a dependency explained in point A. Hope it makes sense. Thanks, galoch On Jul 22, 7:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the
[google-appengine] Re: GAE starting unnecessary instances
The current behavior makes me suspect we are being prepared for Always- on being replaced completely by the new scheduler knobs. Being able to turn up the number of idle instances does make always-on somewhat redundant, as long as the idle instances stick around for a while. Also, if always-on instances WERE being properly utilized right now, it would require artificial load to ascertain the effects of the new scheduler on dynamic instances (assuming the three AO instances were sufficient for your app previously). Many devs wouldn't learn about the new scheduler attributes until after their app (suddenly) becomes popular - not the best time for surprises. And Google wouldn't get as much feedback on the features/behavior of the new scheduler. Is Always-on going to be kept under the new model? /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained
Re: [google-appengine] Re: GAE starting unnecessary instances
I've suspected the same thing. So far we've seen no pricing info for always on and questions about it have went unanswered. Not to mention that $2.10 / week for three always-on instances is a lot different than $40 / week. Robert On Fri, Jul 22, 2011 at 15:46, Tom Phillips tphill0...@gmail.com wrote: The current behavior makes me suspect we are being prepared for Always- on being replaced completely by the new scheduler knobs. Being able to turn up the number of idle instances does make always-on somewhat redundant, as long as the idle instances stick around for a while. Also, if always-on instances WERE being properly utilized right now, it would require artificial load to ascertain the effects of the new scheduler on dynamic instances (assuming the three AO instances were sufficient for your app previously). Many devs wouldn't learn about the new scheduler attributes until after their app (suddenly) becomes popular - not the best time for surprises. And Google wouldn't get as much feedback on the features/behavior of the new scheduler. Is Always-on going to be kept under the new model? /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new
Re: [google-appengine] Re: GAE starting unnecessary instances
After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request while Always On is idle. (Always On instance started) - Incoming request - Always On instance handle the request - another Incoming request (Always On instance busy) - A new Dynamic instance is spawned (Dynamic instance idle, Always on instance busy) - Dynamic instance handle the request - another Incoming request (Dynamic instance idle, Always on instance idle) - Dynamic instance handle the request - No request for more than idle-dynamic-instance-timeout - Dynamic instance shut down - another Incoming request (Always On instance idle) - Always On instance handle the request Hope it makes thing clearer. As part of the new billing model you will have a scheduler knob called 'max-idle-instances' that you can use if extra idling dynamic instances are undesired. The good news is that we are open to suggestion, if you think this behavior is the wrong default, feel free to comment on that thread and I will follow up your suggestion to the Engineering team. On Wed, Jul 20, 2011 at 12:18 AM, Galoch galoch...@gmail.com wrote: Same here. Seems like GAE is totally ignoring Always On instances. I also noticed that even with no user hitting our app and a single cron job that runs every 5 minutes it is still spinning instances every 3 minutes and then killing them in 2 minutes. This has been happening since after the upgrade on 14th July. During peak load this really gets nasty and brings down the performance. This is the feedback I got yesterday from one of our customers since it takes time to spin an instance (and yes we use Spring): 1) I found the GUI to be very laggy Can someone from Google please respond? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
It looks like 2 resident instances and 1 dynamic are stalled as they didn't serve any request for the last minute (check attachment). But still one dynamic instance was started less than 2 minutes ago. Is it normal ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ZS2G6eJC7ioJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. attachment: Instances-VinoCities_2.jpg
[google-appengine] Re: GAE starting unnecessary instances
Even 3 now, for just just a few new requests. Geez. It's getting really expensive :-( -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/xZG_ZE6yDVEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. attachment: Instances-VinoCities_3.jpg
[google-appengine] Re: GAE starting unnecessary instances
Yes, seeing the same problem on all my apps. Java, M/S. Raise a production issue Francois? /Tom On Jul 19, 6:34 am, Francois Masurel f.masu...@gmail.com wrote: Even 3 now, for just just a few new requests. Geez. It's getting really expensive :-( Instances-VinoCities_3.jpg 43KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
http://code.google.com/status/appengine shows they are investigating issue(s) with Java currently.. On Jul 19, 7:28 am, Tom Phillips tphill0...@gmail.com wrote: Yes, seeing the same problem on all my apps. Java, M/S. Raise a production issue Francois? /Tom On Jul 19, 6:34 am, Francois Masurel f.masu...@gmail.com wrote: Even 3 now, for just just a few new requests. Geez. It's getting really expensive :-( Instances-VinoCities_3.jpg 43KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
I'm still seeing this too... On Jul 19, 1:39 pm, Tom Phillips tphill0...@gmail.com wrote: http://code.google.com/status/appengineshows they are investigating issue(s) with Java currently.. On Jul 19, 7:28 am, Tom Phillips tphill0...@gmail.com wrote: Yes, seeing the same problem on all my apps. Java, M/S. Raise a production issue Francois? /Tom On Jul 19, 6:34 am, Francois Masurel f.masu...@gmail.com wrote: Even 3 now, for just just a few new requests. Geez. It's getting really expensive :-( Instances-VinoCities_3.jpg 43KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
Resident instances don't seem to be used at all (check attachment). It's probably a bug on GAE side. Any googlers around ? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/INY70Yliu5YJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. attachment: Instances-VinoCities.jpg
[google-appengine] Re: GAE starting unnecessary instances
Same here. Seems like GAE is totally ignoring Always On instances. I also noticed that even with no user hitting our app and a single cron job that runs every 5 minutes it is still spinning instances every 3 minutes and then killing them in 2 minutes. This has been happening since after the upgrade on 14th July. During peak load this really gets nasty and brings down the performance. This is the feedback I got yesterday from one of our customers since it takes time to spin an instance (and yes we use Spring): 1) I found the GUI to be very laggy Can someone from Google please respond? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
