Hi, I will ask to the engineer team and get back to you in thread.
Hope that helps. On Fri, Jul 22, 2011 at 5:33 PM, Tom Phillips <tphill0...@gmail.com> wrote: > When are threaded instances considered "busy"? > > If it is while they are serving only a one request it would explain > why Always on instances for threaded Java are now severely under- > utilized. > > /Tom > > > > On Jul 22, 10:57 am, Johan Euphrosine <pro...@google.com> wrote: >> HI Galoch, >> >> Thanks for the followup, >> >> I think you are experiencing a combinaison fo the two following rules >> I was pointing to in my previous email: >> (> reads as has priority for handling the incoming request) >> 2/ Spawning a new Dynamic instance > Busy Always On instance >> 4/ Idle Dynamic instance > Idle Always On instance >> >> Applied to your example it could means that: >> Resident Instance 1: Requests: 49 Age: 1Hr >> Resident Instance 2: Requests: 6 Age: 1Hr >> Resident Instance 3: Requests: 2 Age: 1Hr >> Dynamic Instance 1: Requests: 7 Age: 2min >> Dynamic Instance 2: Requests: 291 Age: 1Hr >> Dynamic Instance 3: Requests: 322 Age: 1Hr >> >> - 1 Hours ago while all your Always On instance were busy and you had >> a burst of incoming requests and the scheduler spawned new Dynamic >> instances as per rule 2/ highlighted above. >> - After the burst and back to normal traffic the new Dynamic Instances >> were handing incoming requests in priority as per rule 4/ highlighted >> above. >> - 2 Minutes ago all your instances Always On + Dynamic were busy again >> and the scheduler spawned a new Dynamic instance that handle 7 >> incoming requests. >> >> Hope that make more sense for you and Francois, but as I said earlier >> we are open to suggestion and I will make sure someone working on the >> scheduler team monitor this thread for your input. >> >> >> >> On Fri, Jul 22, 2011 at 9:09 AM, Galoch <galoch...@gmail.com> wrote: >> > @Johan, >> > The issue is not about Always On instance being busy. Its actually the >> > other way ... the Always On instance is never busy ... at least that >> > is what we observed in last 3-4 days. Your explanation may be partly >> > true since this behavior keeps on changing. >> >> > For e.g. I have a snapshot of instances from July 19th and here's the >> > details (for some reason I can't see a link to attach the snapshot >> > images here): >> > Resident Instance 1: Requests: 49 Age: 1Hr >> > Resident Instance 2: Requests: 6 Age: 1Hr >> > Resident Instance 3: Requests: 2 Age: 1Hr >> > Dynamic Instance 1: Requests: 7 Age: 2min >> > Dynamic Instance 2: Requests: 291 Age: 1Hr >> > Dynamic Instance 3: Requests: 322 Age: 1Hr >> >> > This is under "no load" with only very light weight cron jobs running. >> > This gets much much worse during the day under peak load with requests >> > for dynamic instances reaching 1000+ in matter of minutes and resident >> > instances have only "1" request served. >> >> > As you see above Resident Instance 2 and 3 are hardly hit so I don't >> > think they are busy at all. On the other hand, Dynamic Instance 2 and >> > 3 get most of the hits. >> >> > Dynamic Instance 1 is what is killing us. It keeps getting killed and >> > reborn within that 5 minute window!! >> >> > We use Spring framework and it is really very expensive for us when a >> > new instance starts up. >> >> > Just to give you a background, we had gone through a real roller >> > coaster ride to make this to work on GAE by breaking the loading of >> > framework into many different chunks. But still spinning was out of >> > control. Then we found java threads to our rescue. We worked through >> > the hack to load JDO to avoid UnsupportedOperationException. We >> > finally got it to work where most of our requests were served by >> > Always On instances with occasional spinning of Dynamic instances. It >> > was quite impressive. >> >> > Unfortunately, this was short lived when we hit this new behavior with >> > GAE. The very last thing we want GAE to do is create a new instance >> > every few minutes as it could easily reach 30 second deadline during >> > the day and throw critical error. >> >> > I am not sure when the new billing will come into effect but we really >> > need this thing fixed as it literally brings down our app to a >> > grinding halt. So I am open to any suggestions you guys think can help >> > us. >> >> > Another thought about new scheduler is to have a configurable >> > schedule. For e.g. our users are mostly business users who work during >> > normal business hours. We want to be able to spin more Always On >> > instances during those hours and bring the number down during nights >> > and weekends. Dynamic instances won't work for us due to reason >> > explained above. >> >> > Thanks, >> > galoch >> >> > On Jul 21, 5:56 pm, Johan Euphrosine <pro...@google.com> wrote: >> >> After speaking with Engs, I think I can explain what is going on: >> >> >> Here are the current scheduling rules: (> reads as has priority for >> >> handling the incoming request) >> >> >> 1/ Idle Always On instance > Spawning a new Dynamic instance >> >> 2/ Spawning a new Dynamic instance > Busy Always On instance >> >> 3/ Idle Dynamic instance > Busy Always On instance >> >> 4/ Idle Dynamic instance > Idle Always On instance >> >> >> I will give you an example to illustrate the behavior you all noticed, >> >> that is Dynamic instance handling request while Always On is idle. >> >> >> (Always On instance started) >> >> - Incoming request >> >> - Always On instance handle the request >> >> - another Incoming request >> >> (Always On instance busy) >> >> - A new Dynamic instance is spawned >> >> (Dynamic instance idle, Always on instance busy) >> >> - Dynamic instance handle the request >> >> - another Incoming request >> >> (Dynamic instance idle, Always on instance idle) >> >> - Dynamic instance handle the request >> >> - No request for more than idle-dynamic-instance-timeout >> >> - Dynamic instance shut down >> >> - another Incoming request >> >> (Always On instance idle) >> >> - Always On instance handle the request >> >> >> Hope it makes thing clearer. >> >> >> As part of the new billing model you will have a scheduler knob called >> >> 'max-idle-instances' that you can use if extra idling dynamic >> >> instances are undesired. >> >> >> The good news is that we are open to suggestion, if you think this >> >> behavior is the wrong default, feel free to comment on that thread and >> >> I will follow up your suggestion to the Engineering team. >> >> >> On Wed, Jul 20, 2011 at 12:18 AM, Galoch <galoch...@gmail.com> wrote: >> >> > Same here. Seems like GAE is totally ignoring Always On instances. >> >> > I also noticed that even with no user hitting our app and a single >> >> > cron job that runs every 5 minutes it is still spinning instances >> >> > every 3 minutes and then killing them in 2 minutes. >> >> >> > This has been happening since after the upgrade on 14th July. During >> >> > peak load this really gets nasty and brings down the performance. >> >> >> > This is the feedback I got yesterday from one of our customers since >> >> > it takes time to spin an instance (and yes we use Spring): >> >> >> > "1) I found the GUI to be very laggy" >> >> >> > Can someone from Google please respond? >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "Google App Engine" group. >> >> > To post to this group, send email to google-appengine@googlegroups.com. >> >> > To unsubscribe from this group, send email to >> >> > google-appengine+unsubscr...@googlegroups.com. >> >> > For more options, visit this group >> >> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> >> -- >> >> Johan Euphrosine (proppy) >> >> Developer Programs Engineer >> >> Google Developer Relations >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> -- >> Johan Euphrosine (proppy) >> Developer Programs Engineer >> Google Developer Relations > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
