I would suggest placing a strategic pdb.set_trace() around the error
lines right before the bulkloader checks if the error is in the
non_fatal_error_codes set. This way, you could check what the error
code is and add it in the same fashion I did in my patch.
On Feb 2, 3:48 am, antichrist <ttlt...@gmail.com> wrote:
> I have similar problem, but slight different error message.
>
> [ERROR 2011-02-02 14:40:50,286 adaptive_thread_pool.py] Error in
> Thread-1: <urlopen error (10014, 'Bad address')>
> [DEBUG 2011-02-02 14:40:50,301 adaptive_thread_pool.py] Traceback
> (most recent call last):
> File "C:\Documents and Settings\Administrator\My Documents
> \pointmarket\google_appengine\google\appengine\tools\bulkloader.py",
> line 693, in PerformWork
> transfer_time = self._TransferItem(thread_pool)
> File "C:\Documents and Settings\Administrator\My Documents
> \pointmarket\google_appengine\google\appengine\tools\bulkloader.py",
> line 1081, in _TransferItem
> self, retry_parallel=self.first)
> File "C:\Documents and Settings\Administrator\My Documents
> \pointmarket\google_appengine\google\appengine\tools\bulkloader.py",
> line 1358, in GetEntities
> results = self._QueryForPbs(query)
> File "C:\Documents and Settings\Administrator\My Documents
> \pointmarket\google_appengine\google\appengine\tools\bulkloader.py",
> line 1308, in _QueryForPbs
> result_pb)
> File "D:\Program Files\Google\google_appengine\google\appengine\api
> \apiproxy_stub_map.py", line 78, in MakeSyncCall
> return apiproxy.MakeSyncCall(service, call, request, response)
> File "D:\Program Files\Google\google_appengine\google\appengine\api
> \apiproxy_stub_map.py", line 278, in MakeSyncCall
> rpc.CheckSuccess()
> File "D:\Program Files\Google\google_appengine\google\appengine\api
> \apiproxy_rpc.py", line 149, in _WaitImpl
> self.request, self.response)
> File "D:\Program Files\Google\google_appengine\google\appengine\ext
> \remote_api\remote_api_stub.py", line 223, in MakeSyncCall
> handler(request, response)
> File "D:\Program Files\Google\google_appengine\google\appengine\ext
> \remote_api\remote_api_stub.py", line 232, in _Dynamic_RunQuery
> 'datastore_v3', 'RunQuery', query, query_result)
> File "D:\Program Files\Google\google_appengine\google\appengine\ext
> \remote_api\remote_api_stub.py", line 155, in MakeSyncCall
> self._MakeRealSyncCall(service, call, request, response)
> File "D:\Program Files\Google\google_appengine\google\appengine\ext
> \remote_api\remote_api_stub.py", line 167, in _MakeRealSyncCall
> encoded_response = self._server.Send(self._path, encoded_request)
> File "D:\Program Files\Google\google_appengine\google\appengine\tools
> \appengine_rpc.py", line 346, in Send
> f = self.opener.open(req)
> File "D:\Python25\lib\urllib2.py", line 381, in open
> response = self._open(req, data)
> File "D:\Python25\lib\urllib2.py", line 399, in _open
> '_open', req)
> File "D:\Python25\lib\urllib2.py", line 360, in _call_chain
> result = func(*args)
> File "D:\Python25\lib\urllib2.py", line 1107, in http_open
> return self.do_open(httplib.HTTPConnection, req)
> File "D:\Python25\lib\urllib2.py", line 1082, in do_open
> raise URLError(err)
> URLError: <urlopen error (10014, 'Bad address')>
>
> [DEBUG 2011-02-02 14:40:50,301 bulkloader.py] Waiting for
> progress_thread to terminate...
> [DEBUG 2011-02-02 14:40:50,316 bulkloader.py] [Thread-11]
> ExportProgressThread: exiting
> [DEBUG 2011-02-02 14:40:50,316 bulkloader.py] ... done.
> [INFO 2011-02-02 14:40:50,332 bulkloader.py] Have 892 entities, 0
> previously transferred
> [INFO 2011-02-02 14:40:50,332 bulkloader.py] 892 entities (5134783
> bytes) transferred in 63.6 seconds
>
> ---------------------------------------
>
> This "URLError: <urlopen error (10014, 'Bad address')>" error occur
> every time I try to download data after some time.
>
> Has anyone know about this problem?
>
> On 1월24일, 오후5시23분, Ricardo Bánffy <rban...@gmail.com> wrote:
>
> > in case it affects anyone else, one line fixes it:
>
> > Index: google/appengine/tools/bulkloader.py
> > ===================================================================
> > --- google/appengine/tools/bulkloader.py (revision 142)
> > +++ google/appengine/tools/bulkloader.py (working copy)
> > @@ -698,6 +698,8 @@
> > transfer_time)
> > sys.stdout.write('.')
> > sys.stdout.flush()
> > + # Since we had at least one successful transfer, we could
> > assume DNS errors are transient
> > + non_fatal_error_codes.add(-2)
> > status = adaptive_thread_pool.WorkItem.SUCCESS
> > if transfer_time <= MAXIMUM_INCREASE_DURATION:
> > instruction = adaptive_thread_pool.ThreadGate.INCREASE
>
> > BTW, is there a nice constant for this kind of error? Adding a -2 to
> > the set seems dirty, to say the least.
>
> > 2011/1/22 Ricardo Bánffy <rban...@gmail.com>:
>
> > > Hi.
>
> > > I have been, for the past couple days, to download data from the live
> > > app to my local development copy. Every time, sometimes a couple hours
> > > and gigabytes into the download, I get a
>
> > > .................[INFO ] An error occurred. Shutting down...
> > > ..[ERROR ] Error in Thread-8: <urlopen error (-2, 'Name or service
> > > not known')>
>
> > > [INFO ] Have 210 entities, 0 previously transferred
> > > [INFO ] 210 entities (2145028 bytes) transferred in 54.4 seconds
>
> > > message.
>
> > > I assume a try/except with a couple retries around it would fix the
> > > problem - as it looks like a transient DNS failure - and I'll dig
> > > deeper into appcfg.py's code tomorrow in order to fix this, but,
> > > before I do, has anyone fixed this before?
>
> > > BTW, I'm running 1.4.1 on Ubuntu 10.10 with a built-from-sources
> > > Python 2.5.5 (Ubuntu doesn't package it anymore).
>
> > --
> > Ricardo Bánffyhttp://www.dieblinkenlights.comhttp://twitter.com/rbanffy-원본
> > 텍스트 숨기기 -
>
> > - 원본 텍스트 보기 -
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
