Yes, trying to guess about these things is quite unpleasant. Things like pending ms and throttle codes should be clearly documented, especially in light of the upcoming pricing changes.
As I recall, you might see throttle codes + pending ms if your requests need over 1000ms to run or if your requests run very fast relative to your app startup time (I think the logic is that it is better to wait than it is to spin up a new instance). I don't recall which throttle codes indicate each problem, but judging from your log entries I'd suggest making the requests run faster. Robert On Jun 23, 2011 6:03 PM, "Waleed Abdulla" <wal...@ninua.com> wrote: > Thanks Tom, > I'm looking for other solutions as well. I tried a couple of ideas with > backends, but I don't like that I have to manage instances. I loved the > vision of GAE from the start on the promise that I only need to write my > code the right way and then never worry about scaling. > > Now I find myself spending a lot of time trying to trick GAE into > launching more instances to handle my load. No problem with scaling the > code, it's just that GAE doesn't want to give me enough power to get the > work done fast enough. I'm guessing they changed the scaling algorithm last > week. > > I changed my code to run several chains of tasks in parallel. That seems > to help. The hard part is not knowing how the scaling algorithm works, so I > end up re-writing my code many times to figure out which arrangement of > tasks gets GAE to give me enough resources. > > Waleed > > > > > On Tue, Jun 21, 2011 at 10:34 PM, Tom Phillips <tphill0...@gmail.com> wrote: > >> Hi Waleed, >> >> Because I've found tasks to be sometimes unreliable and unpredictable >> performance wise, I've implemented a strategy of engaging backend >> instance(s) as alternative non-task-based job processors. >> >> Basically, I now have two job processing engines that work differently >> and together offer redundancy and better performance. One (the >> traditional one) uses tasks, and the other doesn't - and I run them >> simultaneously. If one isn't working well for some reason, then at >> least the other is getting work done. In my case reliability is more >> of an issue than achieving very high throughput, but this could also >> work to increase overall throughput. >> >> It may not work quite as well for python as it does for java where I >> can have multiple worker threads running on each backend. And it isn't >> necessarily cheap. But it can pick up some slack for the task >> infrastructure, and if the cost allows you could always add more >> backend instances. With threadsafe in java land, it is even better >> since the backend is more fully utilized with multiple worker threads >> working on jobs simultaneously. >> >> Basically the implementation is to have one or more backend worker >> threads/instances that loop eternally looking look for jobs in the DB, >> grabbing batches of them and running them directly (with a short >> configurable pause if none are found in a cycle). Meanwhile, the other >> task-based processor is being driven by a cron job, which every minute >> essentially runs the same "find me jobs" code the backends use. >> Instead of running the jobs directly in the frontend, this one >> dispatches them out as tasks using DB cursors and task chaining to >> handle larger numbers of waiting tasks. >> >> It can create some contention on "job" records in the DB, though there >> should be ways you could minimize that if it becomes excessive. And as >> long as you mark newly found jobs as "assigned" within the same >> transaction that you query them out, you can avoid accidentally >> running the same job twice. Careful if you're on high-replication - >> you may need to account for consistency limitations while querying and >> assigning jobs (I'm on M/S). >> >> /Tom >> >> >> >> On Jun 21, 8:36 pm, Waleed Abdulla <wal...@ninua.com> wrote: >> > Thank you, Nick. >> > I did check your post before asking and pending_ms was missing. Also, >> > I'm noticing "throttle_code=2" in the log of some tasks! It would be >> great >> > to cover that one as well. >> > >> > I've spent the last 3 days trying everything I can think of to make >> my >> > app scale up, but GAE is working against me. I make the tasks longer, and >> > GAE starts running them less often. I make them shorter, and my queues >> start >> > to have a backlog. GAE simply doesn't want to scale up for me (started 3 >> > days ago). I'm running out of ideas to try. Would appreciate your help. >> > >> > More details about what I tried here: >> > >> > http://groups.google.com/group/google-appengine/browse_thread/thread/. .. >> > >> > Regards, >> > Waleed >> > >> > On Tue, Jun 21, 2011 at 5:04 PM, Nick Johnson (Google) < >> > >> > nick.john...@google.com> wrote: >> > > Hi Waleed, >> > >> > > You may find this blog post useful for explaining the various fields of >> log >> > > entries: >> > > http://blog.notdot.net/2011/06/Demystifying-the-App-Engine-request-logs >> > >> > > pending_ms is the amount of time the request spent waiting on an app >> server >> > > to become available to execute the task. >> > >> > > -Nick Johnson >> > >> > > On Wed, Jun 22, 2011 at 8:45 AM, Waleed Abdulla <wal...@ninua.com> >> wrote: >> > >> > >> Good catch, thanks. I'm guessing that pending_ms is the time a request >> > >> stays on hold before execution? I couldn't find it documented >> anywhere. If >> > >> so, then the question is: why would a request be in a pending state? >> Is it >> > >> waiting for an instance to be free? >> > >> > >> This might also be related to my other thread about the task queue not >> > >> scaling up [1]. >> > >> > >> Thanks, >> > >> Waleed >> > >> > >> [1] >> > >>http://groups.google.com/group/google-appengine/browse_thread/thread/ . >> .. >> > >> > >> On Tue, Jun 21, 2011 at 3:33 PM, Nicholas Verne <nve...@google.com >> >wrote: >> > >> > >>> One other note: >> > >> > >>> In a large number of cases, if you the pending_ms (when it exists) to >> > >>> your measurement, you come close to the ms value you circled at the >> > >>> top. >> > >> > >>> Nick Verne >> > >> > >>> On Wed, Jun 22, 2011 at 8:18 AM, Nicholas Verne <nve...@google.com> >> > >>> wrote: >> > >>> > Waleed, >> > >> > >>> > I'm going to have to ask one of the other infrastructure engineers >> > >>> > about the log lines. The value you have circled (18386ms) is not >> > >>> > always greater than the cpu_ms. I'm not sure what this value >> > >>> > indicates. >> > >> > >>> > Nick Verne >> > >> > >>> > On Wed, Jun 22, 2011 at 6:27 AM, Waleed Abdulla <wal...@ninua.com > >> > >>> wrote: >> > >>> >> Nicholas, >> > >>> >> I implemented the change you suggested (making task_t0) a >> global >> > >>> >> variable, but that didn't change the calculations. I'm attaching >> > >>> another >> > >>> >> screen shot. There is a big difference between the actual >> processing >> > >>> time >> > >>> >> and the time reported in the logs. It doesn't happen in all tasks, >> but >> > >>> it's >> > >>> >> often enough that it's slowing down my application considerably >> and >> > >>> causing >> > >>> >> a lot of user complaints. >> > >>> >> I also calculated the total time to load the module (including >> > >>> time for >> > >>> >> imports). It doesn't seem to take much time (sub 1 second), and it >> > >>> happens >> > >>> >> only once right after I deploy a new version. So that is not an >> issue. >> > >>> >> I've set the processing rate, bucket size, and max concurrency >> on >> > >>> the >> > >>> >> task queue to high numbers (much higher than I need). >> > >>> >> Any explanation for why this is happening? >> > >>> >> Waleed >> > >> > >>> >> On Mon, Jun 20, 2011 at 6:54 PM, Brandon Wirtz < >> drak...@digerat.com> >> > >>> wrote: >> > >> > >>> >>> Use a fetch from one thread to another. >> > >> > >>> >>> Echo Time >> > >>> >>> Fetch >> > >>> >>> Echo Time >> > >>> >>> Do something >> > >>> >>> Echo Time >> > >>> >>> Echo Fetch >> > >> > >>> >>> Now you have a timeline of everything except the latency from >> User to >> > >>> >>> Appengine. >> > >> > >>> >>> -----Original Message----- >> > >>> >>> From: google-appengine@googlegroups.com >> > >>> >>> [mailto:google-appengine@googlegroups.com] On Behalf Of Nicholas >> > >>> Verne >> > >>> >>> Sent: Monday, June 20, 2011 6:03 PM >> > >>> >>> To: google-appengine@googlegroups.com >> > >>> >>> Subject: Re: [google-appengine] Why a big difference between >> > >>> execution >> > >>> >>> time >> > >>> >>> and response time? >> > >> > >>> >>> Yes, that's true. If you try this: >> > >> > >>> >>> task_t0 = None >> > >> > >>> >>> ... >> > >> > >>> >>> def main(argv): >> > >>> >>> global task_t0 >> > >>> >>> task_t0 = time.time() >> > >> > >>> >>> You'll get a measurement per request as early as we can make it. >> For >> > >>> >>> warming >> > >>> >>> requests, it might still be worth measuring the import time >> > >> > >>> >>> import time >> > >>> >>> import_start = time.time() >> > >>> >>> ... other imports >> > >>> >>> import_duration = time.time() - import_start >> > >> > >>> >>> The import_duration global should persist and tell you how long >> this >> > >>> >>> instance's imports took to process. Subsequent requests on that >> > >>> instance >> > >>> >>> don't do the imports. >> > >> > >>> >>> Nick Verne >> > >> > >>> >>> On Tue, Jun 21, 2011 at 10:49 AM, Waleed Abdulla < >> wal...@ninua.com> >> > >>> wrote: >> > >>> >>> >> A more useful measurement might be to set task_t0 as a global >> > >>> >>> >> variable at the top of your file, with only the import of >> "time" >> > >>> >>> >> preceding it >> > >> > >>> >>> > Wouldn't that cause the 2nd, 3rd, ...nth requests to the same >> > >>> handler >> > >>> >>> > to show wrong values? I see why setting task_t0 at the top of >> the >> > >>> file >> > >>> >>> > would apply if it was the first request that causes the handler >> to >> > >>> be >> > >>> >>> > loaded. But this is happening on an ongoing basis for all >> requests, >> > >>> >>> > not only the first for that handler. >> > >> > >>> >>> > -- >> > >>> >>> > You received this message because you are subscribed to the >> Google >> > >>> >>> > Groups "Google App Engine" group. >> > >>> >>> > To post to this group, send email to >> > >>> google-appengine@googlegroups.com. >> > >>> >>> > To unsubscribe from this group, send email to >> > >>> >>> > google-appengine+unsubscr...@googlegroups.com. >> > >>> >>> > For more options, visit this group at >> > >>> >>> >http://groups.google.com/group/google-appengine?hl=en. >> > >> > >>> >>> -- >> > >>> >>> You received this message because you are subscribed to the >> Google >> > >>> Groups >> > >>> >>> "Google App Engine" group. >> > >>> >>> To post to this group, send email to >> > >>> google-appengine@googlegroups.com. >> > >>> >>> To unsubscribe from this group, send email to >> > >>> >>> google-appengine+unsubscr...@googlegroups.com. >> > >>> >>> For more options, visit this group at >> > >>> >>>http://groups.google.com/group/google-appengine?hl=en. >> > >> > >>> >>> -- >> > >>> >>> You received this message because you are subscribed to the >> Google >> > >>> Groups >> > >>> >>> "Google App Engine" group. >> > >>> >>> To post to this group, send email to >> > >>> google-appengine@googlegroups.com. >> > >>> >>> To unsubscribe from this group, send email to >> > >>> >>> google-appengine+unsubscr...@googlegroups.com. >> > >>> >>> For more options, visit this group at >> > >>> >>>http://groups.google.com/group/google-appengine?hl=en. >> > >> > >>> >> -- >> > >>> >> You received this message because you are subscribed to the Google >> > >>> Groups >> > >>> >> "Google App Engine" group. >> > >>> >> To post to this group, send email to >> > >>> google-appengine@googlegroups.com. >> > >>> >> To unsubscribe from this group, send email to >> > >>> >> google-appengine+unsubscr...@googlegroups.com. >> > >>> >> For more options, visit this group at >> > >>> >>http://groups.google.com/group/google-appengine?hl=en. >> > >> > >>> -- >> > >>> You received this message because you are subscribed to the Google >> Groups >> > >>> "Google App Engine" group. >> > >>> To post to this group, send email to >> google-appengine@googlegroups.com. >> > >>> To unsubscribe from this group, send email to >> > >>> google-appengine+unsubscr...@googlegroups.com. >> > >>> For more options, visit this group at >> > >>>http://groups.google.com/group/google-appengine?hl=en. >> > >> > >> -- >> > >> You received this message because you are subscribed to the Google >> Groups >> > >> "Google App Engine" group. >> > >> To post to this group, send email to >> google-appengine@googlegroups.com. >> > >> To unsubscribe from this group, send email to >> > >> google-appengine+unsubscr...@googlegroups.com. >> > >> For more options, visit this group at >> > >>http://groups.google.com/group/google-appengine?hl=en. >> > >> > > -- >> > > Nick Johnson, Developer Programs Engineer, App Engine >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Google App Engine" group. >> > > To post to this group, send email to google-appengine@googlegroups.com >> . >> > > To unsubscribe from this group, send email to >> > > google-appengine+unsubscr...@googlegroups.com. >> > > For more options, visit this group at >> > >http://groups.google.com/group/google-appengine?hl=en. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
