Just to close the loop on my issue: I got some off-list help, Twitter investigated, and it turns out my IP address had been blacklisted in error. The blacklisting is removed, and I'm back in business.
I must say it's nice that I could ask a question on this list, and get pretty much immediate attention from the proper Twitter person, and over a weekend at that. Thanks, John Kalucki, and thanks, Twitter. --Jim DeLaHunt, Vancouver, Canada @jdlh http://jdlh.com/en/pr/twanguages.html Twanguages: a language census of Twitter @twanguages On Nov 15, 6:52 am, John Kalucki <jkalu...@gmail.com> wrote: > There are two levels of blacklisting. One is a temporary band that > resets every few minutes. This one gives you 401 errors. Then there's > an IP black hole that is removed by an operator. Currently the IP > black hole sends a TCP RST, but we might might also null route you. > You can verify an IP block by attempting to connect from a different > network. > > If you provide an account name, I can look through the logs and see > what happened. An IP address can also be helpful. In the absence of > these keys, I can only speculate as to what occurred. > > -John Kaluckihttp://twitter.com/jkalucki > Services, Twitter Inc. > > On Nov 15, 12:54 am, Jim DeLaHunt <jim.delah...@gmail.com> wrote: > > > John: > > > Thanks very much for the reply. > > > On Nov 14, 8:30 pm, John Kalucki <jkalu...@gmail.com> wrote: > > > > This sounds like you were ignoring HTTP error codes and eventually got > > > blacklisted. > > > Consider:http://apiwiki.twitter.com/Streaming-API-Documentation#Connecting > > > Hmm... I was launching single curl requests, making one connection > > then breaking it after max 3 seconds. I would then wait 6 minutes > > before trying to connect again. I didn't record the HTTP result code > > I got back, but it seems that according to Streaming-API- > > Documentation#Connecting I was being tremendously conservative. That > > doc recommends backing off for 10 to 240 seconds on an HTTP error code > > (>200); I always backed off for 360 seconds immediately, whether the > > HTTP error code was good or bad. > > > How would backing off by *more* than the docs call for get me > > blacklisted? > > > > You can tell for sure by turning off --silent and using -v to see > > > what's going on. You should be getting some sort of message back, or > > > absolutely nothing back. Those codes are not HTTP error codes, they > > > must be some curl artifact. > > > Correct, the codes "6" and "52" are defined by curl. > > Seehttp://curl.haxx.se/docs/manpage.html. Using -v and other curl > > options, I see clearly that what I'm getting back is "absolutely > > nothing back": 0 bytes in response to my HTTP query. (That's the > > meaning of the code "52".) > > > For the last 6 hours, I've polled once per hour (once per 3600 > > seconds), and this null response has not changed. > > > The docs don't say how to confirm that I've been blacklisted. Any > > suggestions for how to confirm that? Nor do they say what to do if I > > am in fact blacklisted. They say that the blacklist lasts "an > > indeterminate period of time", so maybe they are implying I should > > just wait and the system will list the blacklist itself. > > > The biggest issue, though, is to understand why I could have become > > blacklisted, when I backed off for 360 seconds after each attempt. > > Because right now, I don't know what I should do differently. > > > Thanks again for the guidance. > > --Jim DeLaHunt, Vancouver, Canada �...@jdlh > > Twanguages: a language census of Twitter > > @twanguageshttp://jdlh.com/en/pr/twanguages.html > > > > Tcpdump is also sometimes useful. > > > > -John Kaluckihttp://twitter.com/jkalucki > > > Services, Twitter Inc. > > > > On Nov 14, 6:13 pm, Jim DeLaHunt <jim.delah...@gmail.com> wrote: > > > > > Am I the only one seeing this? I call the Streaming API 10x/hour. For > > > > the last 23 hours or so, I've been getting bad responses every time. > > > > > I use a cron job to call from the Linux shell: > > > > > curl --user myid:mypassword --silent --fail --max-time 3 --retry > > > > 0http://stream.twitter.com/1/statuses/sample.xml > > > > > and I get usually a curl return code "(52) Empty reply from server", > > > > though sometimes "(6) name lookup timed out". Same thing happens when > > > > I ask for .json instead of .xml. > > > > > The failures started at the rate of 1-2/hour on 2009/11/13 09:00h UTC > > > > (Friday early morning PST), though they became continuous as of > > > > 200/11/14 03:24h UTC (Friday evening PST), and remain continuous. > > > > > Is anyone else calling this API and failing? Or succeeding? in the > > > > last 24 hours? > > > > > Thank you, > > > > --Jim DeLaHunt, Vancouver, Canada �...@jdlh > > > > Twanguages: a language census of Twitter > > > > @twanguageshttp://jdlh.com/en/pr/twanguages.html
