I agree with John that to achieve higher user visible reliability, API requests should be wrapped in a retry loop.
However, PLEASE, PLEASE, PLEASE, do NOT use linear backoff, i.e., subsequent retries are delayed by an amount of time chosen uniformly at random up to the same maximum amount for each retry. This will lead to disasters for all API users as failed API requests, when retried, will tend to bunch up in ever increasing bunches, leading to a higher, not lower, failure rate. You should instead use exponential backoff, where the maximum amount of delay time increases for each retry. Queuing theory and experience indicate that the optimum factor used to increase the maximum delay for each retry should be 2.0. The earliest implementations of both Ethernet and TCP, and I'm sure other protocols, tried linear backoff and experienced this problem in spades. When the backoff was changed to exponential, the problems miraculously went away. Jim Renkel -----Original Message----- From: twitter-development-talk@googlegroups.com [mailto:twitter-development-t...@googlegroups.com] On Behalf Of John Kalucki Sent: Wednesday, September 16, 2009 12:56 To: Twitter Development Talk Subject: [twitter-dev] Re: non json response I've been following the internal dialog on tracking this issue down. Given what we know, I don't think there's anything that you can change to the request parameters to reduce the chances of this happening. >From a given client, the chances of this happening to a request are pretty close to random. Different clients, however, seem from the outside to operate differently, as they tend to routed through different infrastructure. There also may be differences in the quality of the code, especially around how errors are handled. If you want higher reliability, I'd suggest wrapping nearly all network API calls in a retry loop. If you get any sort of error: tcp, http, parser, etc. retry with linear backoff. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 16, 10:23 am, Naveen A <knig...@gmail.com> wrote: > Is there a specific way we can construct our request to mitigate the > non-json response? I have used a few different twitter clients on the > same mobile device and some of them do not seem to be plagued with the > bad data like we are? Does including something in the header help get > us through whatever filter is returning the bad data? > > Maybe the Twitter cookies that are returned back on each request? > Currently, we don't pass them back on subsequent requests because they > shouldn't be necessary, but if it will make some of the bad JSON > responses go away, I'll spend the time to implement it.. > > These bad json responses have been a problem for over a month now and > while I realize it is a difficult problem to track down, the fact > remains that the API is not functioning correctly. > > A response from the twitter team would be greatly appreciated. > > On Sep 13, 6:01 am, Rudifa <rudi.far...@gmail.com> wrote: > > > I just had one non-json response, in the middle of about 10 requests > > made with curl -vvv (other responses were correct) > > > Below are 3 requests and the non-json response bracketted by 2 good > > responses which contain the response time and other logging data. > > > HTH > > Rudi > > > rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvvhttp://twitter.com/users/show/rudifa.json > > * About to connect() to twitter.com port 80 (#0) > > * Trying 168.143.161.20... connected > > * Connected to twitter.com (168.143.161.20) port 80 (#0)> GET /users/show/rudifa.json HTTP/1.1 > > > User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 > > > Host: twitter.com > > > Accept: */* > > > < HTTP/1.1 200 OK > > < Date: Sun, 13 Sep 2009 09:45:23 GMT > > < Server: hi > > < X-RateLimit-Limit: 150 > > < X-Transaction: 1252835123-2408-31139 > > < Status: 200 OK > > < ETag: "df090f6c8147e20ba7fe81315a66b9af" > > < Last-Modified: Sun, 13 Sep 2009 09:45:23 GMT > > < X-RateLimit-Remaining: 124 > > < Content-Type: application/json; charset=utf-8 > > < Pragma: no-cache > > < Content-Length: 1176 > > < Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, > > post-check=0 > > < Expires: Tue, 31 Mar 1981 05:00:00 GMT > > < X-Revision: a62881015b2c2fb6f795bf931bd56bd494f37254 > > < X-RateLimit-Reset: 1252836853 > > < Set-Cookie: lang=en; path=/ > > < Set-Cookie: > > _twitter_sess=BAh7CDoRdHJhbnNfcHJvbXB0MDoHaWQiJWU5OGQyZmU3NWVkY2RhZjhkYT k5% 250ANTBlNTA4OTk0MzhhIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFz > > %250AaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--66931156c75554797fc576876bdec52dc 705 736e; > > domain=.twitter.com; path=/ > > < Vary: Accept-Encoding > > < Connection: close > > < > > * Closing connection #0 > > {"profile_sidebar_border_color":"BDDCAD","description":"Wrote firmware > > for world-class osciloscopes for many years. Now learning iPhone > > programming tricks. Loves > > skiing.","url":null,"screen_name":"rudifa","status": > > {"in_reply_to_status_id":null,"favorited":false,"in_reply_to_user_id":nu ll, "source":"<a > > href=\"http://apiwiki.twitter.com/\"; rel=\"nofollow\">API</ > > a>","created_at":"Thu Sep 10 16:49:49 +0000 > > 2009","in_reply_to_screen_name":null,"id": > > 3890997267,"truncated":false,"text":"De retour de la T\u00eate de > > Parmelan"},"following":null,"verified":false,"profile_text_color":"33333 3", "followers_count": > > 9,"profile_background_image_url":"http://a1.twimg.com/ > > profile_background_images/17762518/ > > DSC01211-63-2.jpeg","created_at":"Thu Apr 30 22:42:35 +0000 > > 2009","notifications":null,"friends_count": > > 29,"profile_link_color":"0084B4","profile_background_tile":false,"favour ite s_count": > > 0,"profile_background_color":"9AE4E8","protected":false,"time_zone":"Ber n", "location":"Geneva","name":"Rudi > > Farkas","profile_sidebar_fill_color":"DDFFCC","id": > > 36797542,"statuses_count":52,"utc_offset": > > 3600,"profile_image_url":"http://a1.twimg.com/profile_images/311858510/ > > me-bsp-6a_normal.png"} > > > rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvvhttp://twitter.com/users/show/rudifa.json > > * About to connect() to twitter.com port 80 (#0) > > * Trying 168.143.161.20... connected > > * Connected to twitter.com (168.143.161.20) port 80 (#0)> GET /users/show/rudifa.json HTTP/1.1 > > > User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 > > > Host: twitter.com > > > Accept: */* > > > < HTTP/1.0 200 OK > > < Connection: Close > > < Pragma: no-cache > > < cache-control: no-cache > > < Refresh: 0.1 > > < Content-Type: text/html; charset=iso-8859-1 > > < > > <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/ > > TR/1999/REC-html401-19991224/strict.dtd"> > > <!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" > > "http://www.w3.org/TR/html4/strict.dtd";> --> > > <HTML> > > <HEAD> > > <META HTTP-EQUIV="Refresh" CONTENT="0.1"> > > <META HTTP-EQUIV="Pragma" CONTENT="no-cache"> > > <META HTTP-EQUIV="Expires" CONTENT="-1"> > > <TITLE></TITLE> > > </HEAD> > > <BODY><P></BODY> > > </HTML> > > * Closing connection #0 > > > rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvvhttp://twitter.com/users/show/rudifa.json > > * About to connect() to twitter.com port 80 (#0) > > * Trying 168.143.161.20... connected > > * Connected to twitter.com (168.143.161.20) port 80 (#0)> GET /users/show/rudifa.json HTTP/1.1 > > > User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 > > > Host: twitter.com > > > Accept: */* > > > < HTTP/1.1 200 OK > > < Date: Sun, 13 Sep 2009 09:45:35 GMT > > < Server: hi > > < X-RateLimit-Limit: 150 > > < X-Transaction: 1252835135-57410-29384 > > < Status: 200 OK > > < ETag: "df090f6c8147e20ba7fe81315a66b9af" > > < Last-Modified: Sun, 13 Sep 2009 09:45:35 GMT > > < X-RateLimit-Remaining: 123 > > < Content-Type: application/json; charset=utf-8 > > < Pragma: no-cache > > < Content-Length: 1176 > > < Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, > > post-check=0 > > < Expires: Tue, 31 Mar 1981 05:00:00 GMT > > < X-Revision: a62881015b2c2fb6f795bf931bd56bd494f37254 > > < X-RateLimit-Reset: 1252836853 > > < Set-Cookie: lang=en; path=/ > > < Set-Cookie: > > _twitter_sess=BAh7CDoRdHJhbnNfcHJvbXB0MDoHaWQiJWY0NDM0ODQ3YWI2NDk5OTg4OT Y3% 250AYWQ1ZjU5Njg3OWIwIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFz > > %250AaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA-- > > d454f0c630e2e82ace19b7056947734184aea7ed; domain=.twitter.com; path=/ > > < Vary: Accept-Encoding > > < Connection: close > > < > > * Closing connection #0 > > {"profile_sidebar_border_color":"BDDCAD","description":"Wrote firmware > > for world-class osciloscopes for many years. Now learning iPhone > > programming tricks. Loves > > skiing.","url":null,"screen_name":"rudifa","status": > > {"in_reply_to_status_id":null,"favorited":false,"in_reply_to_user_id":nu ll, "source":"<a > > href=\"http://apiwiki.twitter.com/\"; rel=\"nofollow\">API</ > > a>","created_at":"Thu Sep 10 16:49:49 +0000 > > 2009","in_reply_to_screen_name":null,"id": > > 3890997267,"truncated":false,"text":"De retour de la T\u00eate de > > Parmelan"},"following":null,"verified":false,"profile_text_color":"33333 3", "followers_count": > > 9,"profile_background_image_url":"http://a1.twimg.com/ > > profile_background_images/17762518/ > > DSC01211-63-2.jpeg","created_at":"Thu Apr 30 22:42:35 +0000 > > 2009","notifications":null,"friends_count": > > 29,"profile_link_color":"0084B4","profile_background_tile":false,"favour ite s_count": > > 0,"profile_background_color":"9AE4E8","protected":false,"time_zone":"Ber n", "location":"Geneva","name":"Rudi > > Farkas","profile_sidebar_fill_color":"DDFFCC","id": > > 36797542,"statuses_count":52,"utc_offset": > > 3600,"profile_image_url":"http://a1.twimg.com/profile_images/311858510/ > > me-bsp-6a_normal.png"} > > > On Sep 12, 6:57 pm, Raffi Krikorian <ra...@twitter.com> wrote: > > > > Hi all. > > > > This is an extremely high priority problem for us, and for me > > > personally, to fix. > > > > If you're having this problem, please free to reach out to me, but > > > please try to include: > > > > * the IP address your request is coming from > > > * the request you're making > > > * the response you got back > > > * the time that the request/response was made/received > > > > (or, a tcpdump or curl output when doing -vvv would be sufficient also) > > > > On Sep 10, 2009, at 10:05 PM, Matthew Ranney <m...@ranney.com> wrote: > > > > > I'm seeing tons of these as well. > > > > > However, I've found that if you follow the suggestion of the META > > > > tag to simply "refresh" in 0.1 seconds if you get this bogus > > > > response, you can hide most of this from users, especially if they > > > > are on a fast network. > > > > > On Thu, Sep 10, 2009 at 2:09 PM, Monica Keller <monica.kel...@gmail.com > > > > > wrote: > > > > > We see this error 75% of the time. Have you guys made an progress on > > > > resolving the issue ? > > > > > On Sep 6, 8:14 pm, archF6 <tylerjpeter...@gmail.com> wrote: > > > > > I am able to consistently reproduce this error. I am making GET > > > > > requests via PHP from IP: 96.30.16.192. I receive the error without > > > > > fail after periods of inactivity lasting 2 hours or more. The > > > > header > > > > > response code is 200. Please let me know if I can provide any > > > > > additional info that might help you diagnose the problem, or if you > > > > > have suggestions about how best to handle. > > > > > > Thanks. > > > > > > On Sep 6, 3:35 pm, Rudifa <rudi.far...@gmail.com> wrote: > > > > > > > I have seen this same http page with empty body > > > > > > <!DOCTYPEhtmlPUBLIC "-//W3C//DTDHTML4.01//EN" "http://www.w3.org/ > > > > > > TR/1999/REC-html401-19991224/strict.dtd"> > > > > > > <!-- <!DOCTYPEHTMLPUBLIC "-//W3C//DTDHTML4.01//EN" > > > > > > "http://www.w3.org/TR/html4/strict.dtd";> --> > > > > > > <HTML> > > > > > > <HEAD> > > > > > > <META HTTP-EQUIV="Refresh" CONTENT="0.1"> > > > > > > <META HTTP-EQUIV="Pragma" CONTENT="no-cache"> > > > > > > <META HTTP-EQUIV="Expires" CONTENT="-1"> > > > > > > <TITLE></TITLE> > > > > > > </HEAD> > > ... > > read more »
