Hi Jon, Great!
everpix-alpha I still see what appears to be the bug now and then, but I don't have a reproducible case just waiting in a task queue anymore. On Sep 7, 1:19 pm, Jon McAlister <jon...@google.com> wrote: > Hi Pol, > > I think I have a change that will fix this, which should be available > once 1.5.4 is pushed out. What is the app-id here? > > > > > > > > On Mon, Sep 5, 2011 at 10:41 AM, Pol <i...@pol-online.net> wrote: > > Hi, > > > Our app settings are as follow: > > - Python +HRD > > - Max Idle Instances: ( 2 ) > > - Min Pending Latency: ( 100ms ) > > As of right now, there are 3 instances alive. > > > Without going too much into details, we have GAE integrated with EC2 > > on which we run remote image processing tools. The tools are called > > directly using HTTP GETs from GAE and they returned their results as > > JSON (with gzip content encoding). > > > There are currently 3 tasks in the processing queue on GAE > > continuously failing: the urlfetch() calls to the EC2 tool reach the > > 10 seconds timeout and bail. What doesn't make sense is that calling > > the EC2 tool directly using curl from random machines succeeds in less > > than 1 second. > > > But here's the trick: under certain circumstances, the EC2 tool will > > call back to GAE (HEAD request that does a single db.get()) to check > > if the image has already been processed and this happens for these 3 > > stuck tasks. > > > If calling the EC2 tool from the command line and curl, we have the > > normal behavior: > > - EC2 tool retrieves image from arbitrary URL and computes a hash > > - EC2 tool does a HEAD call to GAE passing this hash to see if image > > has been already processed > > - If yes, return empty JSON > > - If no, process image and return full JSON > > This takes about 1 second. > > > The exact same call done from GAE produces this behavior: > > - EC2 tool retrieves image from arbitrary URL and computes a hash > > - EC2 tool does a HEAD call to GAE passing this hash to see if image > > has been already processed > > -> HEAD call hangs <--- RE-ENTRANCY / DEADLOCK BUG in GAE > > -> urlfetch() from GAE to EC2 reaches 10 seconds deadline and > > aborts <-- BREAKS DEADLOCK > > -> HEAD call finally returns > > - EC2 tool completes normally > > > GAE logs confirm the bug: > > > HEAD call from EC2 / curl origina > > 2011-09-05 10:19:52.502 /api/has_backing? > > bid=90e794f348ac76520076f5d02bc67c87c8a9185b8d36affe8377e73fe4820703 > > 200 368ms 48cpu_ms 8api_cpu_ms 0kb Everpix-Processor > > > HEAD call from EC2 / GAE origin > > 2011-09-05 10:20:44.670 /api/has_backing? > > bid=90e794f348ac76520076f5d02bc67c87c8a9185b8d36affe8377e73fe4820703 > > 200 9712ms 48cpu_ms 8api_cpu_ms 0kb Everpix-Processor > > 2011-09-05 10:20:44.547 /task/import_photo 500 10348ms 28cpu_ms > > 8api_cpu_ms 0kb AppEngine-Google; (+http://code.google.com/appengine) > > (see how the HEAD /api/has_backing call hangs for almost 10 seconds > > and only returns *after* /task/import_photo and its urlfetch() call to > > EC2 has aborted) > > > And finally, AppStats confirms that it's not the head() Python > > execution itself that's hanging: > > > (1) 2011-09-05 09:16:06.843 "HEAD /api/has_backing? > > bid=3bc4aeb08e01d3ba4bfab3282d2a198984a4fc1fab2ad9d1e8a39ee3cddd14da" > > 200 real=227ms cpu=24ms api=8ms overhead=0ms (1 RPC) > > (2) 2011-09-05 09:15:56.422 "POST /task/import_photo" 500 real=10002ms > > cpu=33ms api=8ms overhead=0ms (3 RPCs) > > (3) 2011-09-05 09:15:49.328 "HEAD /api/has_backing? > > bid=90e794f348ac76520076f5d02bc67c87c8a9185b8d36affe8377e73fe4820703" > > 200 real=297ms cpu=21ms api=8ms overhead=0ms (1 RPC) > > > This issue is currently 100% reproducible. > > > - Pol > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
