Hi all,
I've been struggling to get the Python bulk uploader working reliably
with my Java GAE app.
My bulkuploader.yaml is correct (I think) yet sometimes when I'm
partway through uploading a CSV file the uploader barfs with the error
shown at the bottom of this post. The log viewer in GAE shows the
warning "com.google.apphosting.api.DeadlineExceededException: This
request (a5a74b909963ecc7) started at 2010/11/15 16:28:41.971 UTC and
was still executing at 2010/11/15 16:29:11.422 UTC." - I'm not sure
this is relevant though.
My CSV file is 500,000 lines. I split it up into 15,000-line files but
it's hit and miss as to whether those smaller files upload. I reduced
the number of worker threads, bandwidth limit, batch size, etc, but
that didn't seem to make the uploader more reliable.
I guess I could change my app to use app-assigned IDs for keys so I
don't get duplicate entities when I re-upload a file that's failed
partway through, but it'd be great to know why the bulk uploader
doesn't work out of the box.
Is anyone else having this problem and is there a simple solution or
workaround?
Thanks in advance
Andrew.
[ERROR ] [WorkerThread-1] WorkerThread:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/adaptive_thread_pool.py", line 150, in
WorkOnItems
status, instruction = item.PerformWork(self.__thread_pool)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 691, in PerformWork
transfer_time = self._TransferItem(thread_pool)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 848, in _TransferItem
self.request_manager.PostEntities(self.content)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 1300, in PostEntities
datastore.Put(entities)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/datastore.py", line 291, in Put
'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/datastore.py", line 195, in _MakeSyncCall
rpc.check_success()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/apiproxy_stub_map.py", line 499, in check_success
self.__rpc.CheckSuccess()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
self.request, self.response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/ext/remote_api/remote_api_stub.py", line 224, in
MakeSyncCall
handler(request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/ext/remote_api/remote_api_stub.py", line 350, in
_Dynamic_Put
'datastore_v3', 'Put', put_request, put_response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/ext/remote_api/remote_api_stub.py", line 156, in
MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/ext/remote_api/remote_api_stub.py", line 178, in
_MakeRealSyncCall
raise UnknownJavaServerError("An unknown error has occured in the
"
UnknownJavaServerError: An unknown error has occured in the Java
remote_api handler for this call.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
