Hi Ye, I am using the latest release (r65) downloaded from: https://code.google.com/p/appengine-gcs-client/downloads/list
I've generally been testing with manually scaled backend modules, in which case there is no Exception, they just hang indefinitely. A successful write simply has the following in the log: 1. 2013-09-10 10:30:17.830 /_ah/start 200 3156ms 0kb instance=0 0.1.0.3 - - [10/Sep/2013:02:30:17 -0700] "GET /_ah/start HTTP/1.1" 200 119 - - "0.v1.storagetest.benstoragetest.appspot.com" ms=3156 cpu_ms=606 cpm_usd=0.000013 loading_request=1 instance=0 app_engine_release=1.8.4 2. D2013-09-10 10:30:15.582 sleeping 3. D2013-09-10 10:30:15.810 opened file 4. D2013-09-10 10:30:16.915 data written 5. D2013-09-10 10:30:17.728 file closed 6. I2013-09-10 10:30:17.830 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. For an instance that hangs, the log is terminated when I stop the instances: 1. 2013-09-10 10:46:33.041 /_ah/start 500 978206ms 0kb instance=7 0.1.0.3 - - [10/Sep/2013:02:46:33 -0700] "GET /_ah/start HTTP/1.1" 500 0 - - "7.v1.storagetest.benstoragetest.appspot.com" ms=978207 cpu_ms=866 loading_request=1 exit_code=114 instance=7 app_engine_release=1.8.4 2. D2013-09-10 10:30:16.038 sleeping 3. D2013-09-10 10:33:46.377 opened file 4. I2013-09-10 10:46:33.040 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. 5. E2013-09-10 10:46:33.040 Process terminated because the backend took too long to shutdown. If it is useful, here is an example of the failure of a frontend instance: 1. 2013-09-10 10:51:41.971 /gcs?suffix=15 500 60704ms 0kb Wget/1.13.4 (linux-gnu) 137.222.108.37 - - [10/Sep/2013:02:51:41 -0700] "GET /gcs?suffix=15 HTTP/1.1" 500 0 - "Wget/1.13.4 (linux-gnu)" "benstoragetest.appspot.com" ms=60705 cpu_ms=1280 loading_request=1 app_engine_release=1.8.4 instance=00c61b117c1bed3e63c87fa8b71e8c142c32b7 <https://appengine.google.com/instances?app_id=s~benstoragetest&version_id=v1.370120547795114149&key=00c61b117c1bed3e63c87fa8b71e8c142c32b7#00c61b117c1bed3e63c87fa8b71e8c142c32b7> 2. D2013-09-10 10:50:44.196 opened file 3. E2013-09-10 10:51:41.944 Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle result = handler(dict(self._environ), self._StartResponse) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__ rv = self.router.dispatch(request, response) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher return route.handler_adapter(request, response) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__ return handler.dispatch() File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch return method(*args, **kwargs) File "/base/data/home/apps/s~benstoragetest/v1.370120547795114149/storagetest.py", line 26, in get gcs_file.write("f" * 1024 * 1024 * 1 + '\n') File "/base/data/home/apps/s~benstoragetest/v1.370120547795114149/cloudstorage/storage_api.py", line 578, in write self._flush() File "/base/data/home/apps/s~benstoragetest/v1.370120547795114149/cloudstorage/storage_api.py", line 659, in _flush self._send_data(''.join(buffer), last) File "/base/data/home/apps/s~benstoragetest/v1.370120547795114149/cloudstorage/storage_api.py", line 676, in _send_data self._path_with_token, payload=data, headers=headers) File "/base/data/home/apps/s~benstoragetest/v1.370120547795114149/cloudstorage/rest_api.py", line 41, in sync_wrapper return future.get_result() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 325, in get_result self.check_success() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 320, in check_success self.wait() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 304, in wait if not ev.run1(): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 219, in run1 delay = self.run0() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 181, in run0 callback(*args, **kwds) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 375, in _help_tasklet_along ds_conn = datastore._GetConnection() File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/datastore.py", line 391, in _GetConnection return _thread_local.connection_stack[-1] DeadlineExceededError 4. I2013-09-10 10:51:41.971 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application. I have also had a friend replicate the issue (with manually scaled backend modules at least) using the code I posted. Thanks for any suggestions, Ben On Tuesday, September 10, 2013 9:12:36 PM UTC+1, Ye Yuan wrote: > > Hi Ben, > > Thanks for the detailed post. What does the application log for those > hanged request look like? Do they all only have DeadlineExceededException? > What's the stacktrace? > > I don't see anything you are doing wrong. Just make sure you are using a > relatively recent release of the library. > > On Tuesday, September 10, 2013 6:14:58 AM UTC-4, Ben Smithers wrote: >> >> Hi, >> >> I've been having some problems writing data to google cloud storage using >> the python client library ( >> https://developers.google.com/appengine/docs/python/googlecloudstorageclient/ >> ). >> >> Frequently, the write to cloudstorage will 'hang' indefinitely. The call >> to open a new file is successful, but the write (1MB in size in this test >> case) never returns and the file never appears in the bucket. For example, >> I launch 10 instances of a (backend) module, each of which attempts to >> write a file. Typically, somewhere between 4 and 9 of these will succeed, >> with the others hanging after opening. This is the code I am running: >> >> class StartHandler(webapp2.RequestHandler): >> >> GCS_BUCKET="/" >> >> def debugMessage(self, msg): >> logging.debug(msg) >> logservice.flush() >> >> def get(self): >> suffix = str(backends.get_instance()) >> filename=self.GCS_BUCKET + "/testwrite" + suffix + ".txt" >> gcs_file = cloudstorage.open(filename, 'w', content_type= >> 'text/plain' ) >> self.debugMessage("opened file") >> gcs_file.write("f" * 1024 * 1024 * 1 + '\n') >> self.debugMessage("data written") >> gcs_file.close() >> self.debugMessage("file closed") >> >> >> I have also attached a tarred example of the full application in case it >> is relevant (to run, you should only need to modify the application name in >> the two .yaml files and the bucket name in storagetest.py). A few >> additional things: >> >> 1.) I wondered if this was a problem with simultaneous writes so I had >> each instance sleep for 30 seconds * its instance number; I observe the >> same behaviour. >> 2.) I have seen this behaviour on frontend instances, but far far more >> rarely. I modified the above to run in response to a user request - once >> out of 60 times the write hung after opening (a Deadline Exceeded Exception >> was then thrown). >> 3.) I have experimented with the RetryParams (though according to the >> documentation, the defaults should be sufficient) but to no avail. I also >> find it hard to believe this is the issue - I would assume I would be >> getting a TimeoutError. >> >> Has anyone else observed this behaviour? Does anyone have any suggestions >> for what I am doing wrong? Or a different approach to try? >> >> Very grateful for any help, >> Ben >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
