I made the change right before I sent the email. Let me know how it works for you.
Jason, I also made the change to your application. Please report back after tomorrow if you continue to experience issues. On Tue, Dec 15, 2009 at 11:39 AM, Dave Peck <davep...@gmail.com> wrote: > Ikai, > > We'll keep an eye on our app for the next ~24 hours and report back. > > At what time did you make the changes to our instance? We had > substantial downtime earlier today, alas. > > Can you provide any details about what sort of change was made? > > Thanks, > Dave > > On Dec 15, 11:26 am, "Ikai L (Google)" <ika...@google.com> wrote: > > Dave, > > > > You're correct that this is likely affecting other applications, but it's > > not a global issue. There are hotspots in the cloud that we notice are > being > > especially impacted during certain times of the day. We're actively > working > > on addressing these issues, but in the meantime, there are manual steps > we > > can try to prevent your applications from becoming resource starved. We > do > > these on a one-off basis and reserve them only for applications that seem > to > > exhibit the behavior of seeing DeadlineExceeded on simple actions (not > > initial JVM startup), and at fairly predictable intervals during the day. > > I've taken these steps to try to remedy your application. Can you let us > > know if these seem to help? If not, they may indicate that something is > > going on with your application code, though that does not seem like the > case > > here. > > > > > > > > > > > > On Tue, Dec 15, 2009 at 10:54 AM, Dave Peck <davep...@gmail.com> wrote: > > > Hi Ikai, > > > > > Any further details on your end? I get the feeling we're not the only > > > ones, and we've experienced very serious downtime in the last ~48 > > > hours. > > > > > This is a critical issue for us to resolve, but at the same time we > > > lack key pieces of data that would help us solve it on our own... > > > > > Thanks, > > > Dave > > > > > On Dec 15, 9:14 am, Jason C <jason.a.coll...@gmail.com> wrote: > > > > Ikai, > > > > > > We see daily DeadlineExceededErrors on app id 'steprep' from 6.30am > to > > > > 7.30am (log time). > > > > > > Can you look into that as well? > > > > > > Thanks, > > > > j > > > > > > On Dec 14, 3:32 pm, "Ikai L (Google)" <ika...@google.com> wrote: > > > > > > > Do you see that it's consistent at the same times? What's your > > > application > > > > > ID? I'll look into it. > > > > > > > On Mon, Dec 14, 2009 at 11:28 AM, Dave Peck <davep...@gmail.com> > > > wrote: > > > > > > Hello, > > > > > > > > I have an app (citygoround.org) that, especially in the morning, > > > often > > > > > > has 10-15 minutes of outright downtime due to server errors. > > > > > > > > Looking into it, I see that right before the downtime starts, a > few > > > > > > requests log the following warning message: > > > > > > > > > Request was aborted after waiting too long to attempt to > service > > > > > > your request. > > > > > > > Most likely, this indicates that you have reached your > > > > > > simultaneous dynamic request limit. > > > > > > > > I'm certainly not over my limit, but I can believe that the > request > > > in > > > > > > question could take a while. (I'll get to the details of that > request > > > > > > in a moment.) > > > > > > > > Immediately after these warnings, my app has a large amount of > time > > > > > > (10+ minutes) where *all requests* -- no matter how unthreatening > -- > > > > > > raise a DeadlineExceededError. Usually this is raised during the > > > > > > import of an innocuous module like "re" or "time" or perhaps a > Django > > > > > > 1.1 module. (We use use_library.) > > > > > > > > My best theory at the moment is that: > > > > > > > > 1. It's a cold start, so nothing is cached. > > > > > > 2. App Engine encounters the high latency request and bails. > > > > > > 3. We probably inadvertently catch the DeadlineExceededError, so > the > > > > > > runtime doesn't clean up properly. > > > > > > 4. Future requests are left in a busted state. > > > > > > > > Does this sound at all reasonable? I see a few related issues > (2396, > > > > > > 2266, and 1409) but no firm/completely clear discussion of what's > > > > > > happening in any of them. > > > > > > > > Thanks, > > > > > > Dave > > > > > > > > PS: > > > > > > > > The specifics about our high latency request are *not* strictly > > > > > > relevant to the larger problem I'm having, but I will include > them > > > > > > because I have a second "side" question to ask about it. > > > > > > > > The "high latency" request is serving an image. Our app lets > users > > > > > > upload images and we store them in the data store. When serving > an > > > > > > image, our handler: > > > > > > > > 1. Checks to see if the bytes for the image are in memcache, and > if > > > so > > > > > > returns them immediately. > > > > > > 2. Otherwise grabs the image from the datastore, and if it is > smaller > > > > > > than 64K, adds the bytes to the memcache > > > > > > 3. Returns the result > > > > > > > > I'm wondering if using memcache in this way is a smart idea -- it > may > > > > > > very well be the cause of our latency issues. It's hard to tell. > > > > > > > > Alternatively, the issue could be: we have a page that shows a > large > > > > > > number (~100) of such images. If someone requests this page, we > may > > > > > > have a lot of simultaneous image-producing requests happening at > the > > > > > > same time. Perhaps _this_ is the root cause of the original > "Request > > > > > > was aborted" issue? > > > > > > > > Just not sure here... > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > Google > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > > google-appeng...@googlegroups.com. > > > > > > To unsubscribe from this group, send email to > > > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com><google-appengine%2Bunsubscrib > > > e...@googlegroups.com> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > Ikai Lan > > > > > Developer Programs Engineer, Google App Engine > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appengine@googlegroups.com > . > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
