John: Thanks very much for the reply.
On Nov 14, 8:30 pm, John Kalucki <jkalu...@gmail.com> wrote: > This sounds like you were ignoring HTTP error codes and eventually got > blacklisted. > Consider:http://apiwiki.twitter.com/Streaming-API-Documentation#Connecting Hmm... I was launching single curl requests, making one connection then breaking it after max 3 seconds. I would then wait 6 minutes before trying to connect again. I didn't record the HTTP result code I got back, but it seems that according to Streaming-API- Documentation#Connecting I was being tremendously conservative. That doc recommends backing off for 10 to 240 seconds on an HTTP error code (>200); I always backed off for 360 seconds immediately, whether the HTTP error code was good or bad. How would backing off by *more* than the docs call for get me blacklisted? > You can tell for sure by turning off --silent and using -v to see > what's going on. You should be getting some sort of message back, or > absolutely nothing back. Those codes are not HTTP error codes, they > must be some curl artifact. Correct, the codes "6" and "52" are defined by curl. See http://curl.haxx.se/docs/manpage.html . Using -v and other curl options, I see clearly that what I'm getting back is "absolutely nothing back": 0 bytes in response to my HTTP query. (That's the meaning of the code "52".) For the last 6 hours, I've polled once per hour (once per 3600 seconds), and this null response has not changed. The docs don't say how to confirm that I've been blacklisted. Any suggestions for how to confirm that? Nor do they say what to do if I am in fact blacklisted. They say that the blacklist lasts "an indeterminate period of time", so maybe they are implying I should just wait and the system will list the blacklist itself. The biggest issue, though, is to understand why I could have become blacklisted, when I backed off for 360 seconds after each attempt. Because right now, I don't know what I should do differently. Thanks again for the guidance. --Jim DeLaHunt, Vancouver, Canada @jdlh Twanguages: a language census of Twitter @twanguages http://jdlh.com/en/pr/twanguages.html > Tcpdump is also sometimes useful. > > -John Kaluckihttp://twitter.com/jkalucki > Services, Twitter Inc. > > On Nov 14, 6:13 pm, Jim DeLaHunt <jim.delah...@gmail.com> wrote: > > > Am I the only one seeing this? I call the Streaming API 10x/hour. For > > the last 23 hours or so, I've been getting bad responses every time. > > > I use a cron job to call from the Linux shell: > > > curl --user myid:mypassword --silent --fail --max-time 3 --retry > > 0http://stream.twitter.com/1/statuses/sample.xml > > > and I get usually a curl return code "(52) Empty reply from server", > > though sometimes "(6) name lookup timed out". Same thing happens when > > I ask for .json instead of .xml. > > > The failures started at the rate of 1-2/hour on 2009/11/13 09:00h UTC > > (Friday early morning PST), though they became continuous as of > > 200/11/14 03:24h UTC (Friday evening PST), and remain continuous. > > > Is anyone else calling this API and failing? Or succeeding? in the > > last 24 hours? > > > Thank you, > > --Jim DeLaHunt, Vancouver, Canada �...@jdlh > > Twanguages: a language census of Twitter > > @twanguageshttp://jdlh.com/en/pr/twanguages.html
