It looks like there may be an issue with feedsky and responding to "Accepts-Encoding: gzip" header. I changed urlfetch to work like this:
content = urlfetch.fetch(url, headers={ "Accept-Encoding" : "identity" }) It has not failed on me once. I also tried the following experiments. I'm just using straight curl here, no fancy programs. Note how I am cut off sometimes: compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 100k 0 0:00:01 0:00:01 --:--:-- 127k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 133k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 133k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 132k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 132k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 133k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 21 180k 21 39886 0 0 6885 0 0:00:26 0:00:05 0:00:21 0 curl: (18) transfer closed with 145287 bytes remaining to read compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 132k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: gzip" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 21 180k 21 39886 0 0 6879 0 0:00:26 0:00:05 0:00:21 0 curl: (18) transfer closed with 145287 bytes remaining to read Here's the same test passing a different "Accept-Encoding" header: compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 115k 0 0:00:01 0:00:01 --:--:-- 154k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 103k 0 0:00:01 0:00:01 --:--:-- 132k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 115k 0 0:00:01 0:00:01 --:--:-- 155k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 104k 0 0:00:01 0:00:01 --:--:-- 133k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 116k 0 0:00:01 0:00:01 --:--:-- 155k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 116k 0 0:00:01 0:00:01 --:--:-- 153k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 115k 0 0:00:01 0:00:01 --:--:-- 154k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 99141 0 0:00:01 0:00:01 --:--:-- 121k compu...@computer:/tmp$ curl -i -H "Accept-Encoding: identity" http://feed.feedsky.com/qiushi > blah % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 180k 100 180k 0 0 104k 0 0:00:01 0:00:01 --:--:-- 133k Granted, this is a pretty small sample size and not an exhaustive test, but I'd bet money right now that something is up with feedsky.com's GZIP response. On Wed, Nov 18, 2009 at 11:13 PM, Alan Xing <alanx...@gmail.com> wrote: > Hi there, > > We have run into this feed fetch problem, with GAE Python code throwing > DownloadError. This problem seems to happen to a lot of feeds with Chinese > characters inside. Almost any feed from feedsky.com causes such problem. > > http://feed.feedsky.com/qiushi > http://feed.feedsky.com/ihero > http://feed.feedsky.com/ken > http://feed.feedsky.com/cutescr > http://feed.glif.cn/ > > All these feeds can be subscribed correctly by Google Reader. > > Is this a known problem? Any suggestion to fix or work around is > appreciated! > > Thanks, > Alan > > File "D:\Python25\lib\urllib2.py", line 381, in open > > response = self._open(req, data) > > File "D:\Python25\lib\urllib2.py", line 399, in _open > > '_open', req) > > File "D:\Python25\lib\urllib2.py", line 360, in _call_chain > > result = func(*args) > > File "D:\Python25\lib\urllib2.py", line 1107, in http_open > > return self.do_open(httplib.HTTPConnection, req) > > File "D:\Python25\lib\urllib2.py", line 1080, in do_open > > r = h.getresponse() > > File "D:\Program > Files\Google\google_appengine\google\appengine\dist\httplib.py", line > > 203, in getresponse > > self._allow_truncated, self._follow_redirects) > > File "D:\Program > Files\Google\google_appengine\google\appengine\api\urlfetch.py", line > > 241, in fetch > > return rpc.get_result() > > File "D:\Program > > Files\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", > line 478, in > > get_result > > return self.__get_result_hook(self) > > File "D:\Program > Files\Google\google_appengine\google\appengine\api\urlfetch.py", line > > 325, in _get_fetch_result > > raise DownloadError(str(err)) > > DownloadError: ApplicationError: 2 > ['\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xec} > > > \xe9S\x1bY\xb2\xef\xe7\xf6_Q\xd7\x13w\xdc\x13m\xd0\x02bk\xa0oO\x8f\xbb_\xbf\xe9\xc5\xd1 > > \xf6\x8c\xa3\xe3\xc6\r\x87\x90JBc!i\xb4\x18s\xe3\xc5\r\t\x10\x88U\xc2fG\x98 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=. > -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.