I've been using App Engine for probably something like 5 years, I have one major app that has been running for 5 years, it's very well polished, and the traffic and behaviour of the app is very predictable *knocking on wood* I have another app that I've been working on for 3 years, it didn't take off yet, the new app is unpredictable in behaviour, it's vast and unthrottled
While the old app has been handling millions of requests without errors and issues, the new one is failing on even the simplest tasks, the logs are filled with TransientError's, instance failures, transaction failures, the whole thing is chaotic The old app has throttled queues and basic tasks, the throughput is well calibrated to complete all the tasks in 5 hours, using optimal amount of instances, the traffic is regular, it eases in and eases out throughout the day (without throttling, the old app was in similar state before) The new app is built to perform, so it's queues have no limits, it trusts App Engine to scale as much as it can Well turns out that trust isn't well placed, App Engine is supposed to scale on it's own, yet when you leave the limits to the App Engine, it fails to perform You might ask: "Why would I use App Engine if I'm going to manually scale the limits myself?" - That's a good question, If you're going to have to adjust the limits and throughput manually while your app grows, you might as well use AWS or a similar more reliable service This is mostly a rant post, but the advice is still solid, one has to manually calibrate the throughput of routines to prevent app spikes, the instance births and deaths should always be eased in and eased out, otherwise various services of app engine fail to perform On the bright side, throttling also reduces the costs significantly, so it's a good idea to always keep an eye on the app and manually calibrate all routines - on the other side, if your app gains additional traffic without your supervision, these routines will hog and halt ------ On a more technical side, some of these errors are: "Request was aborted after waiting too long to attempt to service your request." - they come in 100's - flood the logs - these are taskqueue push tasks, so the error is pretty stupid, if they can't be handled, they should be left in the queue "google.appengine.api.taskqueue.taskqueue.TransientError" - these are from pull queues, there are invisible/untold limits of pull queues, this is also very concerning, because if your app grows, your scale might be bound by these limits, so try not to use pull queues too much "DeadlineExceededError's" - these are pretty random and rare, yet when you run thousands of tasks, you get these in your logs, they might be omitted Transactions errors and anomalies: these used to happen a lot, but I switched to a pull queue based logic to prevent them, now they are replaced by pull queue anomalies (It would have been great if limits and capacities of each service was more transparent, and I really think taskqueues need some eased bucket configurations, things that will help task batches to be executed in an eased manner, currently the only way to achieve this is to put flat and low throughput limits - similarly, same kind of control can be achieved on the instance scheduler level) ------ Also, after 5 years, I gave up on app engine support, during a time we used to get actual support from this google groups, currently it's just random initial replies and no follow ups, so unless you are paying $500 or something monthly for support, don't expect much support, you are on your own to detect the issues and prevent them through experimentation and volunteer help -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/52165d50-21cc-474e-90a8-276ebabf4303%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
