I'm implementing queuing right now. That will hopefully settle things a little. Once that is done I will provide whatever info I can to get this thing sorted.
@Taylor I would like to specifically thank you for getting back to me. Considering the amount of distress calls you guys get, a response time this quick is awesome. I'm hoping queuing would help me stop all the 420 error codes. The way I'll be implementing this is : 1. First 420 error occurs, start adding unfollows to Queue 2. Queue would try to unfollow, if error, defer for 10s 3. Keep doubling the check time till it reaches say 10 minutes 4. Keep trying every 10 minutes I hope this will ease things out and make my app look good to your abuse detection algorithm :) I tried searching but did not get any definite write up. Is it advisable to back off even when there are 502/503/500 error? The reason I ask is because these errors are quite common and if an app really starts backing off on these errors then there would be a lot of lag. But I have a feeling not backing off on these errors is one of the reasons for the 420 error that my apps encountering, though you would be the right person to answer this. -Nischal On Apr 7, 6:14 am, nischalshetty <nischalshett...@gmail.com> wrote: > Another thing I would like to add is along with these errors I also am > getting a lot of 502 and 503 requests. Do they have to do something > with this? > > -N > > On Apr 7, 5:58 am, nischalshetty <nischalshett...@gmail.com> wrote: > > > > > > > > > Even as I wrote this and got my servers back up, the 420 errors > > started all over again :( Please, can you check on your end using my > > app code or something. I can mail you the app id if you want. The app > > is justunfollow.com > > > -N > > > On Apr 7, 5:55 am, nischalshetty <nischalshett...@gmail.com> wrote: > > > > > There may be others on the IP address you are using abusing the system > > > > and > > > > making you suffer as a result. > > > > Probably but then this had happened twice as I said earlier and both > > > the times none of the api's worked until one of you guys removed the > > > ips from the blacklist. > > > > > Do you get a HTTP status code 420 just for unfollow requests or is it > > > > for > > > > all requests? Have you attempted to perform the operation in isolation > > > > from > > > > your servers (not tied to your application business logic, perhaps using > > > > command line tools or Curl, Twurl, etc.)? > > > > I'm getting it for Unfollow requests only. It does not throw errors > > > when I try it individually even on my own server. I had a test url set > > > up and it did not throw error on unfollowing around 200 users. That's > > > the time I made it live for everyone but it started giving the errors > > > in a few minutes of going live again. > > > > > Can you do the following: > > > > > 1) Issue one of the API calls you're trying to make, taking note of the > > > > exact API URL you're executing (tell me which it is) > > > > (may as well at this time verify that you're using the proper URL > > > > structures, including api.twitter.com as the domain "/1/" prepending all > > > > resource URLs to indicate the API version) > > > > 2) When/if you get the the error response, capture the exact HTTP status > > > > code and the raw/exact HTTP body you get in response? (Not as > > > > interpreted by > > > > any library you're using). > > > > I will do this and revert back. I am using the latest version of > > > twitter4j and it uses api.twitter.com as the end point. > > > > I had put the servers off for the last 5-6 hours now. I just turned > > > them back up. I'm hoping the same thing does not start again :( > > > > -N > > > > On Apr 7, 12:44 am, Taylor Singletary <taylorsinglet...@twitter.com> > > > wrote: > > > > > There may be others on the IP address you are using abusing the system > > > > and > > > > making you suffer as a result. > > > > > Do you get a HTTP status code 420 just for unfollow requests or is it > > > > for > > > > all requests? Have you attempted to perform the operation in isolation > > > > from > > > > your servers (not tied to your application business logic, perhaps using > > > > command line tools or Curl, Twurl, etc.)? > > > > > Can you do the following: > > > > > 1) Issue one of the API calls you're trying to make, taking note of the > > > > exact API URL you're executing (tell me which it is) > > > > (may as well at this time verify that you're using the proper URL > > > > structures, including api.twitter.com as the domain "/1/" prepending all > > > > resource URLs to indicate the API version) > > > > 2) When/if you get the the error response, capture the exact HTTP status > > > > code and the raw/exact HTTP body you get in response? (Not as > > > > interpreted by > > > > any library you're using). > > > > > @episod <http://twitter.com/episod> - Taylor Singletary > > > > > On Wed, Apr 6, 2011 at 12:07 PM, nischalshetty > > > > <nischalshett...@gmail.com>wrote: > > > > > > I cooled down my servers for more than 2 hours now. There were no > > > > > activities happening through my app. I turned it back on just a few > > > > > minutes back. > > > > > > The same problem, getting 420 error codes :( Cooling it off again, can > > > > > you do something to get me out of this trouble? > > > > > > -N > > > > > > On Apr 6, 9:07 pm, nischalshetty <nischalshett...@gmail.com> wrote: > > > > > > Though I have so many users, each user has an internal limit of 100 > > > > > > unfollows per day which is well within any sort of limits. That is > > > > > > one > > > > > > of the reasons why I never implemented any queuing mechanism, such > > > > > > short number of unfollows can be done instantly. > > > > > > > Besides, there were no sudden spikes in traffic so this sudden rate > > > > > > limiting baffles me. And automated unfollows are not allowed so I > > > > > > thought queuing wouldn't be a good idea. I'll try to implement > > > > > > something like that but what's the short term solution? Can you do > > > > > > something on your end to remove the rate limit if any that might > > > > > > have > > > > > > been put on my app? > > > > > > > I have always adhered to twitter limits and rules never going out of > > > > > > my way. Please help me with this. > > > > > > > -Nischal > > > > > > > On Apr 6, 9:00 pm, Taylor Singletary <taylorsinglet...@twitter.com> > > > > > > wrote: > > > > > > > > By running on Google App Engine, or any host where you share IP > > > > > addresses > > > > > > > with another service, you're going to be at a disadvantage when it > > > > > comes to > > > > > > > abuse detection -- my advice would be to ensure you have a queue > > > > > > > system > > > > > in > > > > > > > place for API actions you take (really any application servicing > > > > > anywhere > > > > > > > near as many users as you do should have this in place), > > > > > > > especially > > > > > actions > > > > > > > applied in bulk -- you will have times that you need to queue up > > > > > > > your > > > > > > > actions until you can resume making requests -- the best way to > > > > > > > detect > > > > > if > > > > > > > you can resume making requests is by intermittently trying a > > > > > > > single > > > > > item at > > > > > > > the top of the queue once every few seconds, increasing the > > > > > > > duration > > > > > you > > > > > > > wait with every failed request. > > > > > > > > You never want to continue bursting when faced with an error like > > > > > > > this, > > > > > it > > > > > > > would only make your application appear more abusive to the > > > > > metrics-oriented > > > > > > > unfeeling eye of an abuse detection algorithm. > > > > > > > > @episod <http://twitter.com/episod> - Taylor Singletary > > > > > > > > On Wed, Apr 6, 2011 at 8:47 AM, nischalshetty < > > > > > nischalshett...@gmail.com>wrote: > > > > > > > > > Hi, > > > > > > > > > It's my apphttp://justunfollow.comIt'sbeenrunningfrommorethan a > > > > > > > > year now. I did not see any sudden increase in traffic or any > > > > > > > > of that > > > > > > > > sort. It's hosted on the Google Appengine and there were 2 > > > > > > > > cases in > > > > > > > > the entire year when appengine ips were blocked by twitter. > > > > > > > > > I have more than 200,000 users. Your advice would be greatly > > > > > > > > appreciated. I do not make use of any search or trends api. > > > > > > > > > -N > > > > > > > > > On Apr 6, 8:41 pm, Taylor Singletary > > > > > > > > <taylorsinglet...@twitter.com> > > > > > > > > wrote: > > > > > > > > > Unfollowing is not "rate limited" by the API, but we have > > > > > > > > > organic > > > > > limits > > > > > > > > on > > > > > > > > > most write actions service-wide to prevent certain kinds of > > > > > > > > > bursty > > > > > > > > behavior. > > > > > > > > > Best to back off from retrying requests when you get an error > > > > > > > > > like > > > > > this > > > > > > > > for > > > > > > > > > an exponentially increasing amount of time -- first 30 > > > > > > > > > seconds, > > > > > then a > > > > > > > > > minute, then a few minutes, then 10 minutes, etc. > > > > > > > > > > @episod <http://twitter.com/episod> - Taylor Singletary > > > > > > > > > > On Wed, Apr 6, 2011 at 8:31 AM, nischalshetty < > > > > > nischalshett...@gmail.com > > > > > > > > >wrote: > > > > > > > > > > > I am getting this for the Unfollow method. This shouldn't > > > > > > > > > > happen > > > > > for > > > > > > > > > > the unfollow API. Can someone at twitter look into it? > > > > > > > > > > > -N > > > > > > > > > > > On Apr 6, 8:24 pm, nischalshetty <nischalshett...@gmail.com> > > > > > wrote: > > > > > > > > > > > 420:Returned by the Search and Trends API when you are > > > > > > > > > > > being > > > > > rate > > > > > > > > > > > limited (http://dev.twitter.com/pages/rate-limiting). > > > > > > > > > > > Returned by the Streaming API: > > > > > > > > > > > Too many login attempts in a short period of time. > > > > > > > > > > > Running too many copies of the same application > > > > > > > > > > > authenticating > > > > > with > > > > > > > > > > > the same account name. > > > > > > > > > > > > I'm getting the above error. It's in large numbers. Can > > > > > > > > > > > someone > > > > > help? > > > > > > > > > > > -- > > > > > > > > > > Twitter developer documentation and resources: > > > > > > > >http://dev.twitter.com/doc > > > > > > > > > > API updates via Twitter:http://twitter.com/twitterapi > > > > > > > > > > Issues/Enhancements Tracker: > > > > > > > > > >http://code.google.com/p/twitter-api/issues/list > > > > > > > > > > Change your membership to this group: > > > > > > > > > >http://groups.google.com/group/twitter-development-talk > > > > > > > > > -- > > > > > > > > Twitter developer documentation and resources: > > > > >http://dev.twitter.com/doc > > > > > > > > API updates via Twitter:http://twitter.com/twitterapi > > > > > > > > Issues/Enhancements Tracker: > > > > > > > >http://code.google.com/p/twitter-api/issues/list > > > > > > > > Change your membership to this group: > > > > > > > >http://groups.google.com/group/twitter-development-talk > > > > > > -- > > > > > Twitter developer documentation and > > > > > resources:http://dev.twitter.com/doc > > > > > API updates via Twitter:http://twitter.com/twitterapi > > > > > Issues/Enhancements Tracker: > > > > >http://code.google.com/p/twitter-api/issues/list > > > > > Change your membership to this group: > > > > >http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
