[google-appengine] Re: Weird Instance Scheduler
Same problem here. For last 2+ weeks, my small application, which usually would not use more than 2 instances, has been using up to 8 instances. There's absolutely no increase in traffic and nothing changed on the application side. On Thursday, August 23, 2012 3:58:44 AM UTC+8, Mos wrote: Does anybody else experience abnormal behavior of the instance-scheduler the last three weeks (the last 7 days it got even worse)? (Java / HRD) Or does anybody has profound knowledge about it? Background: My application is unchanged for weeks, configuration not changed and application's traffic is constant. Traffic: One request per minute from Pingdom and around 200 additional pageviews the day (== around 1500 pageviews the day). The peek is not more then 3-4 request per minute. It's very obvious that one instance should be enough for my application. And that was almost the case the last months! But now GAE creates most of the time 3 instances, whereby on has a long life-time for days and the other ones are restarted around 10 to 30 times the day. Because load request takes between 30s to 40s and requests are waiting for loading instances, there are many request that fail (Users and Pingdom agree: *A request that takes more then a couple of seconds is a failed request!*) Please check the attached screenshots that show the behavior! Note: - Killing instances manually did not help - Idle Instances were ( Automatic – 2 ) . Changing it to whatever didn't not change anything; e.g. like ( Automatic – 4 ) Thanks and Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mkWEWUSZ2xcJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
OK, ready to deploy again, and of course I immediately ran into this issue (along with a bunch of problems related to channels). Here are the logs. Note the 7-second gap between the request that created a new instance, and the immediately preceding instance. Note also that there was no warmup request. I now see in the documentation that warmup requests aren't guaranteed, so I'm not sure adding a warmup handler will actually help me; in any case, I haven't explicitly disabled them. So, yeah. No solution at this point, and random 10s delays are not acceptable. Takashi, can you analyze this and tell me what's going on? 1. 1. 2012-09-21 01:50:03.465 /api/game/80001/submit?messages=MOVE+Gray.Cross+600+136+135+134+2+false;MOVE+Gray.Queue+600+34+35+36+0+falseindex=21 200 176ms 0kb Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B206 76.102.149.245 - - [21/Sep/2012:01:50:03 -0700] POST /api/game/80001/submit?messages=MOVE+Gray.Cross+600+136+135+134+2+false;MOVE+Gray.Queue+600+34+35+36+0+falseindex=21 HTTP/1.1 200 192 - Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B206 titan-game-qa.appspot.com ms=176 cpu_ms=163 cpm_usd=0.80 instance=00c61b117c15f0ab71ada0d4f932c37adf981007 https://appengine.google.com/instances?app_id=titan-game-qaversion_id=1.361909218547818204key=00c61b117c15f0ab71ada0d4f932c37adf981007#00c61b117c15f0ab71ada0d4f932c37adf981007 2. 1. 2012-09-21 01:49:50.460 /api/game/80001/submit?messages=SPLIT+Gray.Queue+Gray.Cross+Angel+Centaur+Centaur+Gargoyleindex=17 200 9609ms 0kb Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B206 76.102.149.245 - - [21/Sep/2012:01:49:50 -0700] POST /api/game/80001/submit?messages=SPLIT+Gray.Queue+Gray.Cross+Angel+Centaur+Centaur+Gargoyleindex=17 HTTP/1.1 200 187 - Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B206 titan-game-qa.appspot.com ms=9609 cpu_ms=5063 cpm_usd=0.80 loading_request=1 instance=00c61b117cf26825e763cc7957e0c35cc39d68 https://appengine.google.com/instances?app_id=titan-game-qaversion_id=1.361909218547818204key=00c61b117cf26825e763cc7957e0c35cc39d68#00c61b117cf26825e763cc7957e0c35cc39d68 2. I2012-09-21 01:49:50.460 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This requ 3. 1. 2012-09-21 01:49:33.240 /api/game/80001/submit?messages=MARKERCHOICE+Black.Hornindex=14200 459ms 0kb Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9A405 76.102.149.245 - - [21/Sep/2012:01:49:33 -0700] POST /api/game/80001/submit?messages=MARKERCHOICE+Black.Hornindex=14 HTTP/1.1 200 136 - Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9A405 titan-game-qa.appspot.com ms=459 cpu_ms=187 cpm_usd=0.74 instance=00c61b117c15f0ab71ada0d4f932c37adf981007 https://appengine.google.com/instances?app_id=titan-game-qaversion_id=1.361909218547818204key=00c61b117c15f0ab71ada0d4f932c37adf981007#00c61b117c15f0ab71ada0d4f932c37adf981007 4. 1. 2012-09-21 01:49:31.124 /api/game/80001/submit?messages=COLORCHOICE+Blackindex=11200 244ms 0kb Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9A405 76.102.149.245 - - [21/Sep/2012:01:49:31 -0700] POST /api/game/80001/submit?messages=COLORCHOICE+Blackindex=11 HTTP/1.1 200 129 - Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9A405 titan-game-qa.appspot.com ms=244 cpu_ms=210 cpm_usd=0.73 instance=00c61b117c15f0ab71ada0d4f932c37adf981007 https://appengine.google.com/instances?app_id=titan-game-qaversion_id=1.361909218547818204key=00c61b117c15f0ab71ada0d4f932c37adf981007#00c61b117c15f0ab71ada0d4f932c37adf981007 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bruR2wS3IGoJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
PS. This is with default application settings, and for this test I reverted to using front ends instead of backends (since I gather backends don't support channels, that's no longer an option for me). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-1SggFlu3_sJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
On Mon, Sep 3, 2012 at 7:52 PM, Mos mosa...@googlemail.com wrote: The last three days the GAE instance scheduler works accurate again. There are just 1 or 2 loading-requests the day. Remember: The weeks before I head hundreds of loading-request and many DeadlineExceededException. Hi Mos, But no status-update from Google on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? I just updated the issue with more details. What has happened? - Just luck ? - Did some GAE infrastructure / policy changed on 1th September ? - Did somebody fix the weird instance scheduler? Perhaps after his or her summer holiday ? - Or did praying help ? Please Google, be transparent on this issue. I had a downtime of 5h 19m in August (99,29%) in August. As we discussed before what you refer as downtime is actually a percentage of (loading) requests taking more than 5 seconds (and indenfied by *pingdom* as downtime). The App Engine SLA doesn't have the same definition of downtime https://developers.google.com/appengine/sla Downtime means more than a ten percent Error Rate for any Eligible Application. Downtime Period means, for an Application, a period of five consecutive minutes of Downtime. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods. Error rate for the Service is defined with the Covered Services. As of today, the App Engine SLA only covers the following components, https://developers.google.com/appengine/sla_error_rate. - Serving Infrastructure (HTTP Request sent to App Engine that results in an INTERNAL_SERVING_ERROR) - Datastore (Datastore api call returning one of the following errors: INTERNAL_ERROR, TIMEOUT, BIGTABLE_ERROR, COMMITTED_BUT_STILL_APPLYING, TRY_ALTERNATE_BACKEND ) And unfortunately loading request latency is not covered by the SLA yet. The serving infrastructure team is constantly working on improving the reliability of loading requests performance, but this is a long term effort and in the meantime we (and the App Engine community) can help you to optimize the performance on your application. Other people had similar experiences. We need to know if reliability of GAE is fixed durably! Thanks Mos On Fri, Aug 31, 2012 at 4:51 PM, Mos mosa...@googlemail.com wrote: Today again hundreds of useless instance restarts and many DeadlineExceededException. (Tried many configuration issues. Nothing helps. My last try : max-idle-instance to one) Any news on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? As Kris wrote below the problem exists now for one month! GOOGLE, PLEASE FIX THE CRITICAL PRODUCTION PROBLEM .. we are loosing every day money and customer as long as GAE/Java works like junk !! I deeply regret to trust Google/GAE and build our application for this PaaS. Mos On Fri, Aug 31, 2012 at 3:48 AM, Kristopher Giesing kris.gies...@gmail.com wrote: I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.g...@gmail.com wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-a...@googlegroups.com. To unsubscribe from this group, send email to google-appengi...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at
Re: [google-appengine] Re: Weird Instance Scheduler
Hello Johan, http://code.google.com/p/googleappengine/issues/detail?id=8004 ? I just updated the issue with more details. Thanks, but can you please be a bit more specific? What does The reliability team performed a maintenance operation, and it seems that most application are back to a normal levels of loading requests. mean? - Could this happen from time to time again? - The issue last for weeks! Why do we need to escalate such a critical issue with a 50+ mailing-thread and why isn't GAE able to monitor such bad behavior by itself? - Please Google, be transparent on this issue. I had a downtime of 5h 19m in August (99,29%) in August. As we discussed before what you refer as downtime is actually a percentage of (loading) requests taking more than 5 seconds (and indenfied by *pingdom* as downtime). That's not correct! Pingdom reports downtime if a request takes more than 30 seconds. The Google SLA needs a reality-check! If my users had over and over again to wait more than 30 seconds (in sum over 5h in August) , partly with a DeadlineExceededException, it isn't acceptable! Cheers Mos On Tue, Sep 4, 2012 at 11:53 AM, Johan Euphrosine pro...@google.com wrote: On Mon, Sep 3, 2012 at 7:52 PM, Mos mosa...@googlemail.com wrote: The last three days the GAE instance scheduler works accurate again. There are just 1 or 2 loading-requests the day. Remember: The weeks before I head hundreds of loading-request and many DeadlineExceededException. Hi Mos, But no status-update from Google on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? I just updated the issue with more details. What has happened? - Just luck ? - Did some GAE infrastructure / policy changed on 1th September ? - Did somebody fix the weird instance scheduler? Perhaps after his or her summer holiday ? - Or did praying help ? Please Google, be transparent on this issue. I had a downtime of 5h 19m in August (99,29%) in August. As we discussed before what you refer as downtime is actually a percentage of (loading) requests taking more than 5 seconds (and indenfied by *pingdom* as downtime). The App Engine SLA doesn't have the same definition of downtime https://developers.google.com/appengine/sla Downtime means more than a ten percent Error Rate for any Eligible Application. Downtime Period means, for an Application, a period of five consecutive minutes of Downtime. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods. Error rate for the Service is defined with the Covered Services. As of today, the App Engine SLA only covers the following components, https://developers.google.com/appengine/sla_error_rate. - Serving Infrastructure (HTTP Request sent to App Engine that results in an INTERNAL_SERVING_ERROR) - Datastore (Datastore api call returning one of the following errors: INTERNAL_ERROR, TIMEOUT, BIGTABLE_ERROR, COMMITTED_BUT_STILL_APPLYING, TRY_ALTERNATE_BACKEND ) And unfortunately loading request latency is not covered by the SLA yet. The serving infrastructure team is constantly working on improving the reliability of loading requests performance, but this is a long term effort and in the meantime we (and the App Engine community) can help you to optimize the performance on your application. Other people had similar experiences. We need to know if reliability of GAE is fixed durably! Thanks Mos On Fri, Aug 31, 2012 at 4:51 PM, Mos mosa...@googlemail.com wrote: Today again hundreds of useless instance restarts and many DeadlineExceededException. (Tried many configuration issues. Nothing helps. My last try : max-idle-instance to one) Any news on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? As Kris wrote below the problem exists now for one month! GOOGLE, PLEASE FIX THE CRITICAL PRODUCTION PROBLEM .. we are loosing every day money and customer as long as GAE/Java works like junk !! I deeply regret to trust Google/GAE and build our application for this PaaS. Mos On Fri, Aug 31, 2012 at 3:48 AM, Kristopher Giesing kris.gies...@gmail.com wrote: I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM,
Re: [google-appengine] Re: Weird Instance Scheduler
On Tue, Sep 4, 2012 at 1:27 PM, Mos mosa...@googlemail.com wrote: Hello Johan, http://code.google.com/p/googleappengine/issues/detail?id=8004 ? I just updated the issue with more details. Hi Mos, I updated the issue again. Thanks, but can you please be a bit more specific? What does The reliability team performed a maintenance operation, and it seems that most application are back to a normal levels of loading requests. mean? - Could this happen from time to time again? App Engine is a managed service, the reliability team is constantly monitoring the overall platform health and performing maintenance operation. Google I/O 2011: Life in App Engine Production is a great resource if you are looking for more details about the reliability team operation: http://www.youtube.com/watch?v=rgQm1KEIIuc - The issue last for weeks! Why do we need to escalate such a critical issue with a 50+ mailing-thread and why isn't GAE able to monitor such bad behavior by itself? See my comments on the bug. - Please Google, be transparent on this issue. I had a downtime of 5h 19m in August (99,29%) in August. As we discussed before what you refer as downtime is actually a percentage of (loading) requests taking more than 5 seconds (and indenfied by *pingdom* as downtime). That's not correct! Pingdom reports downtime if a request takes more than 30 seconds. The Google SLA needs a reality-check! You're welcome to make constructive comments and suggestion about the SLA by filling feature requests on the public issue tracker or by contacting premier customer support: http://code.google.com/p/googleappengine/issues/entry?template=Feature%20request https://developers.google.com/appengine/docs/premier/ If my users had over and over again to wait more than 30 seconds (in sum over 5h in August) , partly with a DeadlineExceededException, it isn't acceptable! As I said before the serving infrastructure team is currently working on improving loading request reliability, see: http://code.google.com/p/googleappengine/issues/detail?id=7706 Cheers Mos On Tue, Sep 4, 2012 at 11:53 AM, Johan Euphrosine pro...@google.com wrote: On Mon, Sep 3, 2012 at 7:52 PM, Mos mosa...@googlemail.com wrote: The last three days the GAE instance scheduler works accurate again. There are just 1 or 2 loading-requests the day. Remember: The weeks before I head hundreds of loading-request and many DeadlineExceededException. Hi Mos, But no status-update from Google on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? I just updated the issue with more details. What has happened? - Just luck ? - Did some GAE infrastructure / policy changed on 1th September ? - Did somebody fix the weird instance scheduler? Perhaps after his or her summer holiday ? - Or did praying help ? Please Google, be transparent on this issue. I had a downtime of 5h 19m in August (99,29%) in August. As we discussed before what you refer as downtime is actually a percentage of (loading) requests taking more than 5 seconds (and indenfied by *pingdom* as downtime). The App Engine SLA doesn't have the same definition of downtime https://developers.google.com/appengine/sla Downtime means more than a ten percent Error Rate for any Eligible Application. Downtime Period means, for an Application, a period of five consecutive minutes of Downtime. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods. Error rate for the Service is defined with the Covered Services. As of today, the App Engine SLA only covers the following components, https://developers.google.com/appengine/sla_error_rate. - Serving Infrastructure (HTTP Request sent to App Engine that results in an INTERNAL_SERVING_ERROR) - Datastore (Datastore api call returning one of the following errors: INTERNAL_ERROR, TIMEOUT, BIGTABLE_ERROR, COMMITTED_BUT_STILL_APPLYING, TRY_ALTERNATE_BACKEND ) And unfortunately loading request latency is not covered by the SLA yet. The serving infrastructure team is constantly working on improving the reliability of loading requests performance, but this is a long term effort and in the meantime we (and the App Engine community) can help you to optimize the performance on your application. Other people had similar experiences. We need to know if reliability of GAE is fixed durably! Thanks Mos On Fri, Aug 31, 2012 at 4:51 PM, Mos mosa...@googlemail.com wrote: Today again hundreds of useless instance restarts and many DeadlineExceededException. (Tried many configuration issues. Nothing helps. My last try : max-idle-instance to one) Any news on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? As Kris wrote below the problem exists now for one month! GOOGLE, PLEASE FIX THE CRITICAL PRODUCTION PROBLEM .. we are loosing every day money
Re: [google-appengine] Re: Weird Instance Scheduler
The last three days the GAE instance scheduler works accurate again. There are just 1 or 2 loading-requests the day. Remember: The weeks before I head hundreds of loading-request and many DeadlineExceededException. But no status-update from Google on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? What has happened? - Just luck ? - Did some GAE infrastructure / policy changed on 1th September ? - Did somebody fix the weird instance scheduler? Perhaps after his or her summer holiday ? - Or did praying help ? Please Google, be transparent on this issue. I had a downtime of 5h 19m in August (99,29%) in August. Other people had similar experiences. We need to know if reliability of GAE is fixed durably! Thanks Mos On Fri, Aug 31, 2012 at 4:51 PM, Mos mosa...@googlemail.com wrote: Today again hundreds of useless instance restarts and many DeadlineExceededException. (Tried many configuration issues. Nothing helps. My last try : max-idle-instance to one) Any news on http://code.google.com/p/googleappengine/issues/detail?id=8004? As Kris wrote below the problem exists now for one month! GOOGLE, PLEASE FIX THE CRITICAL PRODUCTION PROBLEM .. we are loosing every day money and customer as long as GAE/Java works like junk !! I deeply regret to trust Google/GAE and build our application for this PaaS. Mos On Fri, Aug 31, 2012 at 3:48 AM, Kristopher Giesing kris.gies...@gmail.com wrote: I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.g...@gmail.com wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/_**wh1KzpESLEJhttps://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
The 400ms was measured from the time that the code entered my servlet's get method. I can't be sure anymore since I've rewritten the code since then to use Objectify, but I'm guessing that it did not include the first PMF construction call. If constructing the PMF is what was costing a bunch of time, then I'm guessing that the warmup request was not constructing one, but that it was getting constructed at static init time relative to the actual request. I'll keep an eye on this once I'm ready to deploy again (the rewrite to Objectify came with a bunch of other changes I need to finish before I'm ready for real testing again). For the moment, though, it seems like the problems I was having were due to a misunderstanding of how GAE instance warmups happen, and not due to a problem with the instance scheduler itself. - Kris On Saturday, September 1, 2012 3:57:07 PM UTC-7, Jeff Schnitzer wrote: Yeah, baffling. JDO startup costs come with the construction of the PersistenceManagerFactory, so that should be in your code. That 400ms - is that measured from a filter at the outermost layer? An interesting thing to try is to set up a handler for the warmup request which issues an actual query to the datastore. Any query. Jeff On Fri, Aug 31, 2012 at 9:25 PM, Kristopher Giesing kris.g...@gmail.com javascript: wrote: OK. Something just became clearer to me. The requests appear to be tagged with the instance that handles the request. Based on that data, it looks like my request is in fact being handled by the resident instance, not the new dynamic instance. The puzzle then becomes why the request still takes 8s to satisfy when the instance handling it is already warmed, and the in-application logging code (which I didn't post, but trust me on this) is never higher than about 400ms. I had been assuming that the 8s cost was the cost of the new instance spinning up, but the instance tag seems to contradict that. The answer has to be some kind of static initialization cost. Although my app is not very complex, I wonder if this is due to the class path scanning that JDO does. I have since switched to Objectify, but I am actually not very clear on whether that is sufficient to prevent JDO/JPA class path scanning; it seems like I would need to evict the JDO/JPA core code from my application on deployment, but it's far from clear to me how to do that. ... But even that may not really explain this behavior because you would think static initialization costs would be born by the warmup request. So, actually, I am baffled. Any ideas, anyone? - Kris On Friday, August 31, 2012 9:16:16 PM UTC-7, Kristopher Giesing wrote: This is the request that I actually issued, being handled: 2012-07-31 23:08:28.045 /api/game/57002?pretty=true 200 7893ms 11kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 76.102.149.245 - kris [31/Jul/2012:23:08:28 -0700] GET /api/game/57002?pretty=true HTTP/1.1 200 11652 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 titan-game-qa.appspot.com ms=7893 cpu_ms=3520 api_cpu_ms=0 cpm_usd=0.099322 instance=00c61b117c77507e2cfe78a0806d0ca80b52720e These are the *two* preceding warmup requests: ** Dynamic instance warmup ** 2012-07-31 23:08:27.475 /_ah/warmup 200 5873ms 0kb 0.1.0.3 - - [31/Jul/2012:23:08:27 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-game-qa.appspot.com ms=5873 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=00c61b117cdaae6145945d99c16aeee7cc0f4ad8 ** Resident/idle instance warmup ** 2012-07-31 23:07:42.842 /_ah/warmup 200 5045ms 0kb 0.1.0.3 - - [31/Jul/2012:23:07:42 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-game-qa.appspot.com ms=5046 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=00c61b117c77507e2cfe78a0806d0ca80b52720e This is my point. The problem is not that a new instance was spawned (although I admit that I did not quite understand the desired behavior when I first posted this data). The problem is that the request I issued is not satisfied until AFTER the warmup request has been issued and handled by the new instance. The request should FIRST have been handled by the already resident instance, AND THEN the new instance should have been spawned. If I'm misunderstanding something, please clarify, because at the face of it this seems to be a smoking gun. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4FGx8YdHUIgJ.
RE: [google-appengine] Re: Weird Instance Scheduler
So, actually, I am baffled. Any ideas, anyone? Does your warm up load all your classes? Warm is kind of relative J -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
Today again hundreds of useless instance restarts and many DeadlineExceededException. (Tried many configuration issues. Nothing helps. My last try : max-idle-instance to one) Any news on http://code.google.com/p/googleappengine/issues/detail?id=8004 ? As Kris wrote below the problem exists now for one month! GOOGLE, PLEASE FIX THE CRITICAL PRODUCTION PROBLEM .. we are loosing every day money and customer as long as GAE/Java works like junk !! I deeply regret to trust Google/GAE and build our application for this PaaS. Mos On Fri, Aug 31, 2012 at 3:48 AM, Kristopher Giesing kris.gies...@gmail.comwrote: I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.g...@gmail.com wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/_**wh1KzpESLEJhttps://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
On Fri, Aug 31, 2012 at 10:48 AM, Kristopher Giesing kris.gies...@gmail.com wrote: I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. As far as I can see, all of the loading requests in the logs are warmup requests(sorry if I misread the logs). This is very likely an expected behavior. If your resident instance get a request, and if the request is CPU intensive, our scheduler needs to spin up a new instance by sending a warmup request in order to keep the number of idle instance. This will help absorbing subsequent traffic, and this behavior is definitely what the resident instances are for. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. Yeah, if you can file an issue with that information, that will definitely help. However, please keep in mind the expected behavior I mentioned above, and add your expected behavior in detail(don't say just 'It didn't work') alongside the things you actually observe. Thanks, -- Takashi - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.g...@gmail.com wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/_**wh1KzpESLEJhttps://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ubhrxTXYlC4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Takashi Matsuo | Developers Advocate | tmat...@google.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
This is the request that I actually issued, being handled: 2012-07-31 23:08:28.045 /api/game/57002?pretty=true 200 7893ms 11kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 76.102.149.245 - kris [31/Jul/2012:23:08:28 -0700] GET /api/game/57002?pretty=true HTTP/1.1 200 11652 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 titan-game-qa.appspot.com ms=7893 cpu_ms=3520 api_cpu_ms=0 cpm_usd=0.099322 instance=00c61b117c77507e2cfe78a0806d0ca80b52720e These are the *two* preceding warmup requests: ** Dynamic instance warmup ** 2012-07-31 23:08:27.475 /_ah/warmup 200 5873ms 0kb 0.1.0.3 - - [31/Jul/2012:23:08:27 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-game-qa.appspot.com ms=5873 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=00c61b117cdaae6145945d99c16aeee7cc0f4ad8 ** Resident/idle instance warmup ** 2012-07-31 23:07:42.842 /_ah/warmup 200 5045ms 0kb 0.1.0.3 - - [31/Jul/2012:23:07:42 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-game-qa.appspot.com ms=5046 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=00c61b117c77507e2cfe78a0806d0ca80b52720e This is my point. The problem is not that a new instance was spawned (although I admit that I did not quite understand the desired behavior when I first posted this data). The problem is that the request I issued is not satisfied until AFTER the warmup request has been issued and handled by the new instance. The request should FIRST have been handled by the already resident instance, AND THEN the new instance should have been spawned. If I'm misunderstanding something, please clarify, because at the face of it this seems to be a smoking gun. - Kris On Friday, August 31, 2012 8:37:47 AM UTC-7, Takashi Matsuo (Google) wrote: On Fri, Aug 31, 2012 at 10:48 AM, Kristopher Giesing kris.g...@gmail.comjavascript: wrote: I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. As far as I can see, all of the loading requests in the logs are warmup requests(sorry if I misread the logs). This is very likely an expected behavior. If your resident instance get a request, and if the request is CPU intensive, our scheduler needs to spin up a new instance by sending a warmup request in order to keep the number of idle instance. This will help absorbing subsequent traffic, and this behavior is definitely what the resident instances are for. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. Yeah, if you can file an issue with that information, that will definitely help. However, please keep in mind the expected behavior I mentioned above, and add your expected behavior in detail(don't say just 'It didn't work') alongside the things you actually observe. Thanks, -- Takashi - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.g...@gmail.com wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/_**wh1KzpESLEJhttps://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ubhrxTXYlC4J. To post to this group, send email to google-a...@googlegroups.comjavascript: . To
Re: [google-appengine] Re: Weird Instance Scheduler
OK. Something just became clearer to me. The requests appear to be tagged with the instance that handles the request. Based on that data, it looks like my request is in fact being handled by the resident instance, not the new dynamic instance. The puzzle then becomes why the request still takes 8s to satisfy when the instance handling it is already warmed, and the in-application logging code (which I didn't post, but trust me on this) is never higher than about 400ms. I had been assuming that the 8s cost was the cost of the new instance spinning up, but the instance tag seems to contradict that. The answer has to be some kind of static initialization cost. Although my app is not very complex, I wonder if this is due to the class path scanning that JDO does. I have since switched to Objectify, but I am actually not very clear on whether that is sufficient to prevent JDO/JPA class path scanning; it seems like I would need to evict the JDO/JPA core code from my application on deployment, but it's far from clear to me how to do that. ... But even that may not really explain this behavior because you would think static initialization costs would be born by the warmup request. So, actually, I am baffled. Any ideas, anyone? - Kris On Friday, August 31, 2012 9:16:16 PM UTC-7, Kristopher Giesing wrote: This is the request that I actually issued, being handled: 2012-07-31 23:08:28.045 /api/game/57002?pretty=true 200 7893ms 11kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 76.102.149.245 - kris [31/Jul/2012:23:08:28 -0700] GET /api/game/57002?pretty=true HTTP/1.1 200 11652 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 titan-game-qa.appspot.com ms=7893 cpu_ms=3520 api_cpu_ms=0 cpm_usd=0.099322 instance=00c61b117c77507e2cfe78a0806d0ca80b52720e These are the *two* preceding warmup requests: ** Dynamic instance warmup ** 2012-07-31 23:08:27.475 /_ah/warmup 200 5873ms 0kb 0.1.0.3 - - [31/Jul/2012:23:08:27 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-game-qa.appspot.com ms=5873 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=00c61b117cdaae6145945d99c16aeee7cc0f4ad8 ** Resident/idle instance warmup ** 2012-07-31 23:07:42.842 /_ah/warmup 200 5045ms 0kb 0.1.0.3 - - [31/Jul/2012:23:07:42 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-game-qa.appspot.com ms=5046 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=00c61b117c77507e2cfe78a0806d0ca80b52720e This is my point. The problem is not that a new instance was spawned (although I admit that I did not quite understand the desired behavior when I first posted this data). The problem is that the request I issued is not satisfied until AFTER the warmup request has been issued and handled by the new instance. The request should FIRST have been handled by the already resident instance, AND THEN the new instance should have been spawned. If I'm misunderstanding something, please clarify, because at the face of it this seems to be a smoking gun. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4FGx8YdHUIgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
Does your warmup request initialize the persistent manager, or some libraries you may want to preload beforehand? On Sep 1, 2012 1:26 PM, Kristopher Giesing kris.gies...@gmail.com wrote: OK. Something just became clearer to me. The requests appear to be tagged with the instance that handles the request. Based on that data, it looks like my request is in fact being handled by the resident instance, not the new dynamic instance. The puzzle then becomes why the request still takes 8s to satisfy when the instance handling it is already warmed, and the in-application logging code (which I didn't post, but trust me on this) is never higher than about 400ms. I had been assuming that the 8s cost was the cost of the new instance spinning up, but the instance tag seems to contradict that. The answer has to be some kind of static initialization cost. Although my app is not very complex, I wonder if this is due to the class path scanning that JDO does. I have since switched to Objectify, but I am actually not very clear on whether that is sufficient to prevent JDO/JPA class path scanning; it seems like I would need to evict the JDO/JPA core code from my application on deployment, but it's far from clear to me how to do that. ... But even that may not really explain this behavior because you would think static initialization costs would be born by the warmup request. So, actually, I am baffled. Any ideas, anyone? - Kris On Friday, August 31, 2012 9:16:16 PM UTC-7, Kristopher Giesing wrote: This is the request that I actually issued, being handled: 2012-07-31 23:08:28.045 /api/game/57002?pretty=true 200 7893ms 11kb Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 76.102.149.245 - kris [31/Jul/2012:23:08:28 -0700] GET /api/game/57002?pretty=true HTTP/1.1 200 11652 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25 titan-game-qa.appspot.com ms=7893 cpu_ms=3520 api_cpu_ms=0 cpm_usd=0.099322 instance=**00c61b117c77507e2cfe78a0806d0c** a80b52720e These are the *two* preceding warmup requests: ** Dynamic instance warmup ** 2012-07-31 23:08:27.475 /_ah/warmup 200 5873ms 0kb 0.1.0.3 - - [31/Jul/2012:23:08:27 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-**game-qa.appspot.comhttp://1.360723738856412175.titan-game-qa.appspot.com/ ms=5873 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=**00c61b117cdaae6145945d99c16aee**e7cc0f4ad8 ** Resident/idle instance warmup ** 2012-07-31 23:07:42.842 /_ah/warmup 200 5045ms 0kb 0.1.0.3 - - [31/Jul/2012:23:07:42 -0700] GET /_ah/warmup HTTP/1.1 200 60 - - 1.360723738856412175.titan-**game-qa.appspot.comhttp://1.360723738856412175.titan-game-qa.appspot.com/ ms=5046 cpu_ms=2475 api_cpu_ms=0 cpm_usd=0.068778 loading_request=1 instance=**00c61b117c77507e2cfe78a0806d0c**a80b52720e This is my point. The problem is not that a new instance was spawned (although I admit that I did not quite understand the desired behavior when I first posted this data). The problem is that the request I issued is not satisfied until AFTER the warmup request has been issued and handled by the new instance. The request should FIRST have been handled by the already resident instance, AND THEN the new instance should have been spawned. If I'm misunderstanding something, please clarify, because at the face of it this seems to be a smoking gun. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4FGx8YdHUIgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
Hello Takashi, I can offer you to deploy my Spring MVC application to another app-engine-account for testing purpose (Sure, it shouldn't be public available there.) Then you can test and configure it like you want Or even simpler: My application is for days really unreliable. Many requests take more then 20s or 30 seconds (because of loading requests). Hence -- it couldn't get much more worse. You are free to change my configuration settings, as long as we arrange a specific time periods and as long as my bill doesn't go up. (Just contact me directly) Bye the way: Is a colleague of yours evaluation http://code.google.com/p/googleappengine/issues/detail?id=8004? Then we should wait, otherwise his evaluation may be disrupted. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', The problem is not restricted to resident instances (min idle instances). From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. Please check this example: http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 Cheers Mos On Thu, Aug 30, 2012 at 12:54 AM, Takashi Matsuo tmat...@google.com wrote: Hi Mos and everyone, I'm trying to reproduce the issue about min idle instance which some of you guys reported here in this thread, saying Setting min idle instances doesn't work for me. My initial test is just with a simple helloworld Java application multithread enabled, setting 1 min idle instance, and setting 1 min cron job. I ran this test for about 2 and half days. I think it just worked as expected. The resident instance had been alive and handled 3625 requests during the test. What I'm planning to do next is another experiment with an application with Spring MVC. I'll update with the result hopefully next week. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', hopefully with expected behavior, actual results, a characteristic of the application like average time for loading requests as well as normal requests, etc. I've done a quick search on our issue tracker, and I don't think there's any issue yet. If there's already an issue about it, please let me know. Thanks, On Tue, Aug 28, 2012 at 2:13 AM, Carl Schroeder schroeder.car...@gmail.com wrote: Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/**googleappengine/issues/detail?**id=8004#c8http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comwrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost
Re: [google-appengine] Re: Weird Instance Scheduler
I posted a great deal of information in the thread here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/rjZhjMEAXUI In that thread I posted logs that showed that the very first request after setting min instances to 1 will spawn a new instance (in addition to the instance that the min instances setting created). The app ID used in that testing is titan-game-qa and the timestamps are in the logs I posted. At some point I will have enough bandwidth to set up a more specific test, but I feel I've already posted plenty of information for GAE engineers to digest. - Kris On Monday, August 27, 2012 2:17:47 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.g...@gmail.com javascript: wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-a...@googlegroups.comjavascript:. To unsubscribe from this group, send email to google-appengi...@googlegroups.com javascript:. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ubhrxTXYlC4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
Hi Mos and everyone, I'm trying to reproduce the issue about min idle instance which some of you guys reported here in this thread, saying Setting min idle instances doesn't work for me. My initial test is just with a simple helloworld Java application multithread enabled, setting 1 min idle instance, and setting 1 min cron job. I ran this test for about 2 and half days. I think it just worked as expected. The resident instance had been alive and handled 3625 requests during the test. What I'm planning to do next is another experiment with an application with Spring MVC. I'll update with the result hopefully next week. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', hopefully with expected behavior, actual results, a characteristic of the application like average time for loading requests as well as normal requests, etc. I've done a quick search on our issue tracker, and I don't think there's any issue yet. If there's already an issue about it, please let me know. Thanks, On Tue, Aug 28, 2012 at 2:13 AM, Carl Schroeder schroeder.car...@gmail.comwrote: Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/**googleappengine/issues/detail?**id=8004#c8http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comwrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.comwrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic
Re: [google-appengine] Re: Weird Instance Scheduler
This is not a very good test. Better would be: Run 'ab -c 1' against it and see if you get any cold starts. Change 1 to a larger number, up to what concurrency we should expect for a multithreaded instance. Jeff On Wed, Aug 29, 2012 at 6:54 PM, Takashi Matsuo tmat...@google.com wrote: Hi Mos and everyone, I'm trying to reproduce the issue about min idle instance which some of you guys reported here in this thread, saying Setting min idle instances doesn't work for me. My initial test is just with a simple helloworld Java application multithread enabled, setting 1 min idle instance, and setting 1 min cron job. I ran this test for about 2 and half days. I think it just worked as expected. The resident instance had been alive and handled 3625 requests during the test. What I'm planning to do next is another experiment with an application with Spring MVC. I'll update with the result hopefully next week. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', hopefully with expected behavior, actual results, a characteristic of the application like average time for loading requests as well as normal requests, etc. I've done a quick search on our issue tracker, and I don't think there's any issue yet. If there's already an issue about it, please let me know. Thanks, On Tue, Aug 28, 2012 at 2:13 AM, Carl Schroeder schroeder.car...@gmail.com wrote: Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/**googleappengine/issues/detail?**id=8004#c8http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comwrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened.
Re: [google-appengine] Re: Weird Instance Scheduler
Jeff, Thanks for the suggestion, and probably that's true. I've chosen this test from Mos's e-mail, because I got a feeling that he saw odd behaviors even with one request per minute. Hopefully I can do another test based on your suggestion soon. Please note that you can also provide your test result on our issue tracker and help us reproduce the issue :) Thanks, On Thu, Aug 30, 2012 at 9:16 AM, Jeff Schnitzer j...@infohazard.org wrote: This is not a very good test. Better would be: Run 'ab -c 1' against it and see if you get any cold starts. Change 1 to a larger number, up to what concurrency we should expect for a multithreaded instance. Jeff On Wed, Aug 29, 2012 at 6:54 PM, Takashi Matsuo tmat...@google.comwrote: Hi Mos and everyone, I'm trying to reproduce the issue about min idle instance which some of you guys reported here in this thread, saying Setting min idle instances doesn't work for me. My initial test is just with a simple helloworld Java application multithread enabled, setting 1 min idle instance, and setting 1 min cron job. I ran this test for about 2 and half days. I think it just worked as expected. The resident instance had been alive and handled 3625 requests during the test. What I'm planning to do next is another experiment with an application with Spring MVC. I'll update with the result hopefully next week. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', hopefully with expected behavior, actual results, a characteristic of the application like average time for loading requests as well as normal requests, etc. I've done a quick search on our issue tracker, and I don't think there's any issue yet. If there's already an issue about it, please let me know. Thanks, On Tue, Aug 28, 2012 at 2:13 AM, Carl Schroeder schroeder.car...@gmail.com wrote: Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/**googleappengine/issues/detail?** id=8004#c8http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comwrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi
Re: [google-appengine] Re: Weird Instance Scheduler
I really do wish I had time right now to help track this down - believe me, this issue is very relevant to my interests! Jeff On Wed, Aug 29, 2012 at 9:07 PM, Takashi Matsuo tmat...@google.com wrote: Jeff, Thanks for the suggestion, and probably that's true. I've chosen this test from Mos's e-mail, because I got a feeling that he saw odd behaviors even with one request per minute. Hopefully I can do another test based on your suggestion soon. Please note that you can also provide your test result on our issue tracker and help us reproduce the issue :) Thanks, On Thu, Aug 30, 2012 at 9:16 AM, Jeff Schnitzer j...@infohazard.orgwrote: This is not a very good test. Better would be: Run 'ab -c 1' against it and see if you get any cold starts. Change 1 to a larger number, up to what concurrency we should expect for a multithreaded instance. Jeff On Wed, Aug 29, 2012 at 6:54 PM, Takashi Matsuo tmat...@google.comwrote: Hi Mos and everyone, I'm trying to reproduce the issue about min idle instance which some of you guys reported here in this thread, saying Setting min idle instances doesn't work for me. My initial test is just with a simple helloworld Java application multithread enabled, setting 1 min idle instance, and setting 1 min cron job. I ran this test for about 2 and half days. I think it just worked as expected. The resident instance had been alive and handled 3625 requests during the test. What I'm planning to do next is another experiment with an application with Spring MVC. I'll update with the result hopefully next week. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', hopefully with expected behavior, actual results, a characteristic of the application like average time for loading requests as well as normal requests, etc. I've done a quick search on our issue tracker, and I don't think there's any issue yet. If there's already an issue about it, please let me know. Thanks, On Tue, Aug 28, 2012 at 2:13 AM, Carl Schroeder schroeder.car...@gmail.com wrote: Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/**googleappengine/issues/detail?** id=8004#c8http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comwrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance
Re: [google-appengine] Re: Weird Instance Scheduler
Try making a page that consists of more than a single request to the server. A burst of requests that is served (not static content) under the pending latency time would usually trigger an instance spin-up. Now that spin-up times are back to normal, I am not seeing this behavior nearly as often. On Wednesday, August 29, 2012 6:07:30 PM UTC-7, Takashi Matsuo (Google) wrote: Jeff, Thanks for the suggestion, and probably that's true. I've chosen this test from Mos's e-mail, because I got a feeling that he saw odd behaviors even with one request per minute. Hopefully I can do another test based on your suggestion soon. Please note that you can also provide your test result on our issue tracker and help us reproduce the issue :) Thanks, On Thu, Aug 30, 2012 at 9:16 AM, Jeff Schnitzer je...@infohazard.orgjavascript: wrote: This is not a very good test. Better would be: Run 'ab -c 1' against it and see if you get any cold starts. Change 1 to a larger number, up to what concurrency we should expect for a multithreaded instance. Jeff On Wed, Aug 29, 2012 at 6:54 PM, Takashi Matsuo tma...@google.comjavascript: wrote: Hi Mos and everyone, I'm trying to reproduce the issue about min idle instance which some of you guys reported here in this thread, saying Setting min idle instances doesn't work for me. My initial test is just with a simple helloworld Java application multithread enabled, setting 1 min idle instance, and setting 1 min cron job. I ran this test for about 2 and half days. I think it just worked as expected. The resident instance had been alive and handled 3625 requests during the test. What I'm planning to do next is another experiment with an application with Spring MVC. I'll update with the result hopefully next week. At the same time, I'd like one of you to file an issue on our issue tracker for this particular topic, 'Setting min idle instances doesn't work', hopefully with expected behavior, actual results, a characteristic of the application like average time for loading requests as well as normal requests, etc. I've done a quick search on our issue tracker, and I don't think there's any issue yet. If there's already an issue about it, please let me know. Thanks, On Tue, Aug 28, 2012 at 2:13 AM, Carl Schroeder schroede...@gmail.comjavascript: wrote: Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/**googleappengine/issues/detail?** id=8004#c8http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comwrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is
Re: [google-appengine] Re: Weird Instance Scheduler
On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroeder.car...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
On Mon, Aug 27, 2012 at 7:17 AM, Kristopher Giesing kris.gies...@gmail.com wrote: Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. Hi Kristopher, Can you comment with the appid and timestamps of when this last happened? Thanks in advance. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com javascript: wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-a...@googlegroups.comjavascript:. To unsubscribe from this group, send email to google-appengi...@googlegroups.com javascript:. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ApT6E62dU9QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroeder.car...@gmail.comwrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/**G4DPOlW2Jh8Jhttps://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ApT6E62dU9QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Carl D'Halluin Next-generation communication at http://www.rogerthat.net Email: c...@mobicage.com Phone: +32 9 324 25 64 Fax: +32 9 324 25 65 Skype: carldhalluin Twitter: @carl_dhalluin LinkedIn: http://www.linkedin.com/pub/carl-d-halluin/0/982/457 NV MOBICAGE Antwerpsesteenweg 19 9080 Lochristi Belgium -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin c...@mobicage.com wrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroeder.car...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/**G4DPOlW2Jh8Jhttps://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ApT6E62dU9QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Carl D'Halluin Next-generation communication at http://www.rogerthat.net Email: c...@mobicage.com Phone: +32 9 324 25 64 Fax: +32 9 324 25 65 Skype: carldhalluin Twitter: @carl_dhalluin LinkedIn: http://www.linkedin.com/pub/carl-d-halluin/0/982/457 NV MOBICAGE Antwerpsesteenweg 19 9080 Lochristi Belgium -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To
Re: [google-appengine] Re: Weird Instance Scheduler
In http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mosa...@googlemail.com wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin c...@mobicage.com wrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroeder.car...@gmail.com wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/**G4DPOlW2Jh8Jhttps://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ApT6E62dU9QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Carl D'Halluin Next-generation
Re: [google-appengine] Re: Weird Instance Scheduler
Yep. Googlites, let us know what else you need to run this down. On Monday, August 27, 2012 10:05:41 AM UTC-7, Mos wrote: In http://code.google.com/p/googleappengine/issues/detail?id=8004#c8 I described in detail a current example of the nonconforming instance-handling of GAE. Please check the comment, the screenshot and the log-file I filed there. Dear GAE-Team, what else do you need to fix this? In this thread and in several issues you should have more than enough proof and examples. Cheers Mos On Mon, Aug 27, 2012 at 5:34 PM, Mos mos...@googlemail.com javascript:wrote: I saw the same behavior (as discussed before in the thread). Many other people reported this again and again on this mailing-list. Google has to acknowledge that the current implementation is buggy or the implementation works but doesn't make any sense in practice. Bye the way - The problem is not restricted to resident instances. From time to time the same happens for dynamic instances: One or more dynamic instances are running and are almost idle (sometimes really idle==no request or just one request is served). Request comes and starts a new dynamic instance, it goes through 30-40 seconds of warmup, then request is served. On Mon, Aug 27, 2012 at 5:19 PM, Carl D'Halluin ca...@mobicage.comjavascript: wrote: Hi Carl, I see exactly the same behaviour for my Java appengine app. Resident instance does nothing; instead idle instance is started, going through several seconds of warmup, then request is served. Regards On Mon, Aug 27, 2012 at 5:11 PM, Carl Schroeder schroede...@gmail.comjavascript: wrote: 2012-08-27 08:05 is the point in the logs. 1 Resident instance. No Dynamic instances. The request was sent to a cold starting Dynamic instance. Resident instance did nothing. Request took 18 seconds to serve. On Monday, August 27, 2012 2:16:25 AM UTC-7, Johan Euphrosine (Google) wrote: On Mon, Aug 27, 2012 at 5:59 AM, Carl Schroeder schroede...@gmail.com wrote: Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. Hi Carl, That's not what we observed, as I corrected in the previous email: Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. If you observe a different behavior please comment with your application id and the timestamp of occurence and we can try to figure out what happened. Thanks in advance. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/**G4DPOlW2Jh8Jhttps://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@**googlegroups.com. For more options, visit this group at http://groups.google.com/**group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ApT6E62dU9QJ. To post to this group, send email to
[google-appengine] Re: Weird Instance Scheduler
Hi Can somebody explain how it is possible that - I have 1 resident java appengine instance - I didnt send any test requests for 90 minutes - When I sent the request, the system had to warm up, although I have 1 resident java appengine. What does resident exactly mean? My settings are: * Idle Instances: ( 1 – 1 ) * Pending latency: (Automatic - 500ms) Logs (blanked out some stuff with xxx): 2012-08-24 22:14:43.786 /xxx 200 8872ms 0kb AppEngine-Google; (+http://code.google.com/appengine; appid: s~xxx) 0.1.0.40 - - [24/Aug/2012:15:14:43 -0700] POST /xxx HTTP/1.1 200 793 - AppEngine-Google; (+http://code.google.com/appengine; appid: s~xxx) xxx.appspot.com ms=8873 cpu_ms=5157 cpm_usd=0.89 loading_request=1 instance=00c61b117c1360f2e8f3cdce153a3c79777a7e81 I 2012-08-24 22:14:43.786 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. 2012-08-24 20:46:47.635 /xxx 200 1862ms 0kb AppEngine-Google; (+http://code.google.com/appengine) 0.1.0.2 - - [24/Aug/2012:13:46:47 -0700] POST /xxx HTTP/1.1 200 0 xxx AppEngine-Google; (+http://code.google.com/appengine) xxx.appspot.com ms=1863 cpu_ms=260 cpm_usd=0.000260 queue_name=default task_name=9251619801338774487 instance=00c61b117c266e667e6954feaef8bf2492f23006 Thank you! On Wednesday, August 22, 2012 9:58:44 PM UTC+2, Mos wrote: Does anybody else experience abnormal behavior of the instance-scheduler the last three weeks (the last 7 days it got even worse)? (Java / HRD) Or does anybody has profound knowledge about it? Background: My application is unchanged for weeks, configuration not changed and application's traffic is constant. Traffic: One request per minute from Pingdom and around 200 additional pageviews the day (== around 1500 pageviews the day). The peek is not more then 3-4 request per minute. It's very obvious that one instance should be enough for my application. And that was almost the case the last months! But now GAE creates most of the time 3 instances, whereby on has a long life-time for days and the other ones are restarted around 10 to 30 times the day. Because load request takes between 30s to 40s and requests are waiting for loading instances, there are many request that fail (Users and Pingdom agree: *A request that takes more then a couple of seconds is a failed request!*) Please check the attached screenshots that show the behavior! Note: - Killing instances manually did not help - Idle Instances were ( Automatic – 2 ) . Changing it to whatever didn't not change anything; e.g. like ( Automatic – 4 ) Thanks and Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/YT9dk2OEsg8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
On Sun, Aug 26, 2012 at 11:25 PM, Mobicage c...@mobicage.com wrote: Hi Can somebody explain how it is possible that - I have 1 resident java appengine instance - I didnt send any test requests for 90 minutes - When I sent the request, the system had to warm up, although I have 1 resident java appengine. What does resident exactly mean? I believe Jon described resident in great details in this thread: https://groups.google.com/d/msg/google-appengine/nRtzGtG9790/hLS16qux_04J Because the scheduler is now treating the reserved instances as Min-Idle-Instances, what you're describing is expected behavior. They are intentionally kept idle, and it tries to serve traffic using the non-reserved instances. Then, if the non-reserved instances can't keep up, then it will make use of the reserved instances. That is, to repeat, the invariant that the scheduler is trying to maintain here is that your app has at least 3 idle instances. And if an instance is getting traffic, then it isn't idle. The value of an idle instance is that it can process requests right-away if needed, without having to first warmup or do a loading request. It sounds like what you'd really prefer is something like Min-Instances, but that's not presently an available option. The scheduler route traffic to resident idle instance when dynamic instances can't keep with the traffic and always try to keep Min Idle Instances in reserve at all times, I believe it currently need to start at least 1 dynamic instance before making use of the resident idle capacity. My settings are: * Idle Instances: ( 1 – 1 ) * Pending latency: (Automatic - 500ms) Logs (blanked out some stuff with xxx): 2012-08-24 22:14:43.786 /xxx 200 8872ms 0kb AppEngine-Google; (+http://code.google.com/appengine; appid: s~xxx) 0.1.0.40 - - [24/Aug/2012:15:14:43 -0700] POST /xxx HTTP/1.1 200 793 - AppEngine-Google; (+http://code.google.com/appengine; appid: s~xxx) xxx.appspot.com ms=8873 cpu_ms=5157 cpm_usd=0.89 loading_request=1 instance=00c61b117c1360f2e8f3cdce153a3c79777a7e81 I 2012-08-24 22:14:43.786 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. 2012-08-24 20:46:47.635 /xxx 200 1862ms 0kb AppEngine-Google; (+http://code.google.com/appengine) 0.1.0.2 - - [24/Aug/2012:13:46:47 -0700] POST /xxx HTTP/1.1 200 0 xxx AppEngine-Google; (+http://code.google.com/appengine) xxx.appspot.com ms=1863 cpu_ms=260 cpm_usd=0.000260 queue_name=default task_name=9251619801338774487 instance=00c61b117c266e667e6954feaef8bf2492f23006 Thank you! On Wednesday, August 22, 2012 9:58:44 PM UTC+2, Mos wrote: Does anybody else experience abnormal behavior of the instance-scheduler the last three weeks (the last 7 days it got even worse)? (Java / HRD) Or does anybody has profound knowledge about it? Background: My application is unchanged for weeks, configuration not changed and application's traffic is constant. Traffic: One request per minute from Pingdom and around 200 additional pageviews the day (== around 1500 pageviews the day). The peek is not more then 3-4 request per minute. It's very obvious that one instance should be enough for my application. And that was almost the case the last months! But now GAE creates most of the time 3 instances, whereby on has a long life-time for days and the other ones are restarted around 10 to 30 times the day. Because load request takes between 30s to 40s and requests are waiting for loading instances, there are many request that fail (Users and Pingdom agree: A request that takes more then a couple of seconds is a failed request!) Please check the attached screenshots that show the behavior! Note: - Killing instances manually did not help - Idle Instances were ( Automatic – 2 ) . Changing it to whatever didn't not change anything; e.g. like ( Automatic – 4 ) Thanks and Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/YT9dk2OEsg8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
On Mon, Aug 27, 2012 at 12:26 AM, Johan Euphrosine pro...@google.com wrote: On Sun, Aug 26, 2012 at 11:25 PM, Mobicage c...@mobicage.com wrote: Hi Can somebody explain how it is possible that - I have 1 resident java appengine instance - I didnt send any test requests for 90 minutes - When I sent the request, the system had to warm up, although I have 1 resident java appengine. What does resident exactly mean? I believe Jon described resident in great details in this thread: https://groups.google.com/d/msg/google-appengine/nRtzGtG9790/hLS16qux_04J Because the scheduler is now treating the reserved instances as Min-Idle-Instances, what you're describing is expected behavior. They are intentionally kept idle, and it tries to serve traffic using the non-reserved instances. Then, if the non-reserved instances can't keep up, then it will make use of the reserved instances. That is, to repeat, the invariant that the scheduler is trying to maintain here is that your app has at least 3 idle instances. And if an instance is getting traffic, then it isn't idle. The value of an idle instance is that it can process requests right-away if needed, without having to first warmup or do a loading request. It sounds like what you'd really prefer is something like Min-Instances, but that's not presently an available option. The scheduler route traffic to resident idle instance when dynamic instances can't keep with the traffic and always try to keep Min Idle Instances in reserve at all times, I believe it currently need to start at least 1 dynamic instance before making use of the resident idle capacity. Correction, I answered that too fast. Resident instances are used for processing incoming request if there is no dynamic instance, but it is possible that the scheduler warm up new dynamic instance to maintain the Min Idle Instance invariant. carl@ if you can give us your application id, and the last timestamp of occurence of the event you reported, we can try to figure out why your test spawned a new dynamic instance. My settings are: * Idle Instances: ( 1 – 1 ) * Pending latency: (Automatic - 500ms) Logs (blanked out some stuff with xxx): 2012-08-24 22:14:43.786 /xxx 200 8872ms 0kb AppEngine-Google; (+http://code.google.com/appengine; appid: s~xxx) 0.1.0.40 - - [24/Aug/2012:15:14:43 -0700] POST /xxx HTTP/1.1 200 793 - AppEngine-Google; (+http://code.google.com/appengine; appid: s~xxx) xxx.appspot.com ms=8873 cpu_ms=5157 cpm_usd=0.89 loading_request=1 instance=00c61b117c1360f2e8f3cdce153a3c79777a7e81 I 2012-08-24 22:14:43.786 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. 2012-08-24 20:46:47.635 /xxx 200 1862ms 0kb AppEngine-Google; (+http://code.google.com/appengine) 0.1.0.2 - - [24/Aug/2012:13:46:47 -0700] POST /xxx HTTP/1.1 200 0 xxx AppEngine-Google; (+http://code.google.com/appengine) xxx.appspot.com ms=1863 cpu_ms=260 cpm_usd=0.000260 queue_name=default task_name=9251619801338774487 instance=00c61b117c266e667e6954feaef8bf2492f23006 Thank you! On Wednesday, August 22, 2012 9:58:44 PM UTC+2, Mos wrote: Does anybody else experience abnormal behavior of the instance-scheduler the last three weeks (the last 7 days it got even worse)? (Java / HRD) Or does anybody has profound knowledge about it? Background: My application is unchanged for weeks, configuration not changed and application's traffic is constant. Traffic: One request per minute from Pingdom and around 200 additional pageviews the day (== around 1500 pageviews the day). The peek is not more then 3-4 request per minute. It's very obvious that one instance should be enough for my application. And that was almost the case the last months! But now GAE creates most of the time 3 instances, whereby on has a long life-time for days and the other ones are restarted around 10 to 30 times the day. Because load request takes between 30s to 40s and requests are waiting for loading instances, there are many request that fail (Users and Pingdom agree: A request that takes more then a couple of seconds is a failed request!) Please check the attached screenshots that show the behavior! Note: - Killing instances manually did not help - Idle Instances were ( Automatic – 2 ) . Changing it to whatever didn't not change anything; e.g. like ( Automatic – 4 ) Thanks and Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/YT9dk2OEsg8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options,
Re: [google-appengine] Re: Weird Instance Scheduler
Let me see if I understand this correctly: there is currently no way on app engine to ensure that there is an instance ready to process incoming requests for an app that has been idle for some period of time. Min idle instances (labeled as Resident) sit there and do almost nothing while user facing requests are instead sent to cold instance starts. If true, that dovetails with what I have seen in the behavior of my app. For python runtimes with sub-second spinup times, this is no big deal. For java runtimes with spinup times in double digit seconds it is a deal-breaker of a feature. The problem seems to be that the scheduler thinks sending a request to a non-existent dynamic instance is a better idea than using the Resident instance for it's intended purpose: to serve requests when dynamic instances are unable to. This is probably a corner case born of low traffic conditions that allow user request serving dynamic instances to despawn. For low traffic apps, Resident instances serve almost no purpose. Better to do away with them via the slider bars and just set up a script to tickle the app just often enough to keep one Dynamic instance resident. So, two features to fix this: First, a slider bar labeled Minimum Dynamic instances ;) Second, a button to enable sending warm-up requests and having them return before considering an instance for user facing requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/G4DPOlW2Jh8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Weird Instance Scheduler
Resident instances are used for processing incoming request if there is no dynamic instance This is the behavior we all want, but experimentation seems to indicate it doesn't happen, at least for some apps. - Kris -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_wh1KzpESLEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Weird Instance Scheduler
In addition here the failed-request report from Pingdom the last day (that's not acceptable!): al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 3:166 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 3:166 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 3:346 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 3:356 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 17:446 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 17:456 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 17:576 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 17:586 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 18:176 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 18:186 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 18:346 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 18:356 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 18:436 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 18:446 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 19:026 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 19:036 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 19:056 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 19:056 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 19:416 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 19:426 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 19:516 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 19:526 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 20:026 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 20:056 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 20:106 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 20:126 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 23:036 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 23:046 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 23:106 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 23:106 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN Mi 23:156 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UPMi 23:166 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN 02:486 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UP 02:496 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN 03:046 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UP 03:066 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN 03:126 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UP 03:136 KBkrisentalk al...@pingdom.comDOWN alert: krisentalk (www.krisentalk.de) is DOWN 06:256 KBkrisentalk al...@pingdom.comUP alert: krisentalk (www.krisentalk.de) is UP 06:266 KBkrisentalk On Wed, Aug 22, 2012 at 9:58 PM, Mos mosa...@googlemail.com wrote: Does anybody else experience abnormal behavior of the instance-scheduler the last three weeks (the last 7 days it got even worse)? (Java / HRD) Or does anybody has profound knowledge about it? Background: My application is unchanged for weeks, configuration not changed and application's traffic is constant. Traffic: One request per minute from Pingdom and around 200 additional pageviews the day (== around 1500 pageviews the day). The peek is not more then 3-4 request per minute. It's very obvious that one instance should be enough for my application. And that was almost the case the last months! But now GAE creates most of the time 3 instances, whereby on has a long life-time for days and the other ones are restarted around 10 to 30 times the
[google-appengine] Re: Weird Instance Scheduler
I fill a production issue for this. If anybody has similar problems or is interested in it please star it: http://code.google.com/p/googleappengine/issues/detail?id=8004 On Wed, Aug 22, 2012 at 9:58 PM, Mos mosa...@googlemail.com wrote: Does anybody else experience abnormal behavior of the instance-scheduler the last three weeks (the last 7 days it got even worse)? (Java / HRD) Or does anybody has profound knowledge about it? Background: My application is unchanged for weeks, configuration not changed and application's traffic is constant. Traffic: One request per minute from Pingdom and around 200 additional pageviews the day (== around 1500 pageviews the day). The peek is not more then 3-4 request per minute. It's very obvious that one instance should be enough for my application. And that was almost the case the last months! But now GAE creates most of the time 3 instances, whereby on has a long life-time for days and the other ones are restarted around 10 to 30 times the day. Because load request takes between 30s to 40s and requests are waiting for loading instances, there are many request that fail (Users and Pingdom agree: *A request that takes more then a couple of seconds is a failed request!*) Please check the attached screenshots that show the behavior! Note: - Killing instances manually did not help - Idle Instances were ( Automatic – 2 ) . Changing it to whatever didn't not change anything; e.g. like ( Automatic – 4 ) Thanks and Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
