ikai, you bet your money correctly. we confirmed your diagnose. The pain point, though, is that there are quite a few popular feed service providers having these kind of problems. sorry for replying back to you late.
On Sat, Nov 21, 2009 at 3:40 AM, Ikai L (Google) <ika...@google.com> wrote: > It looks like there may be an issue with feedsky and responding to > "Accepts-Encoding: gzip" header. I changed urlfetch to work like this: > > content = urlfetch.fetch(url, headers={ "Accept-Encoding" : "identity" }) > > It has not failed on me once. I also tried the following experiments. I'm > just using straight curl here, no fancy programs. Note how I am cut off > sometimes: > > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 100k 0 0:00:01 0:00:01 --:--:-- > 127k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 133k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 133k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 132k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 132k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 133k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 21 180k 21 39886 0 0 6885 0 0:00:26 0:00:05 > 0:00:21 0 > curl: (18) transfer closed with 145287 bytes remaining to read > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 132k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 21 180k 21 39886 0 0 6879 0 0:00:26 0:00:05 > 0:00:21 0 > curl: (18) transfer closed with 145287 bytes remaining to read > > Here's the same test passing a different "Accept-Encoding" header: > > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 115k 0 0:00:01 0:00:01 --:--:-- > 154k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- > 132k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 115k 0 0:00:01 0:00:01 --:--:-- > 155k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 104k 0 0:00:01 0:00:01 --:--:-- > 133k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 116k 0 0:00:01 0:00:01 --:--:-- > 155k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 116k 0 0:00:01 0:00:01 --:--:-- > 153k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 115k 0 0:00:01 0:00:01 --:--:-- > 154k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 99141 0 0:00:01 0:00:01 --:--:-- > 121k > compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" > http://feed.feedsky.com/qiushi > blah > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 180k 100 180k 0 0 104k 0 0:00:01 0:00:01 --:--:-- > 133k > > Granted, this is a pretty small sample size and not an exhaustive test, but > I'd bet money right now that something is up with feedsky.com's GZIP > response. > > > On Wed, Nov 18, 2009 at 11:13 PM, Alan Xing <alanx...@gmail.com> wrote: > >> Hi there, >> >> We have run into this feed fetch problem, with GAE Python code throwing >> DownloadError. This problem seems to happen to a lot of feeds with Chinese >> characters inside. Almost any feed from feedsky.com causes such problem. >> >> http://feed.feedsky.com/qiushi >> http://feed.feedsky.com/ihero >> http://feed.feedsky.com/ken >> http://feed.feedsky.com/cutescr >> http://feed.glif.cn/ >> >> All these feeds can be subscribed correctly by Google Reader. >> >> Is this a known problem? Any suggestion to fix or work around is >> appreciated! >> >> Thanks, >> Alan >> >> File "D:\Python25\lib\urllib2.py", line 381, in open >> >> response = self._open(req, data) >> >> File "D:\Python25\lib\urllib2.py", line 399, in _open >> >> '_open', req) >> >> File "D:\Python25\lib\urllib2.py", line 360, in _call_chain >> >> result = func(*args) >> >> File "D:\Python25\lib\urllib2.py", line 1107, in http_open >> >> return self.do_open(httplib.HTTPConnection, req) >> >> File "D:\Python25\lib\urllib2.py", line 1080, in do_open >> >> r = h.getresponse() >> >> File "D:\Program >> Files\Google\google_appengine\google\appengine\dist\httplib.py", line >> >> 203, in getresponse >> >> self._allow_truncated, self._follow_redirects) >> >> File "D:\Program >> Files\Google\google_appengine\google\appengine\api\urlfetch.py", line >> >> 241, in fetch >> >> return rpc.get_result() >> >> File "D:\Program >> >> Files\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", >> line 478, in >> >> get_result >> >> return self.__get_result_hook(self) >> >> File "D:\Program >> Files\Google\google_appengine\google\appengine\api\urlfetch.py", line >> >> 325, in _get_fetch_result >> >> raise DownloadError(str(err)) >> >> DownloadError: ApplicationError: 2 >> ['\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xec} >> >> >> \xe9S\x1bY\xb2\xef\xe7\xf6_Q\xd7\x13w\xdc\x13m\xd0\x02bk\xa0oO\x8f\xbb_\xbf\xe9\xc5\xd1 >> >> >> \xf6\x8c\xa3\xe3\xc6\r\x87\x90JBc!i\xb4\x18s\xe3\xc5\r\t\x10\x88U\xc2fG\x98 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=. >> > > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.