Re: [twitter-dev] Re: Search API rate limit change?

[Hayes Davis]

I just wanted to add to this. The 420s have let up for the most part and I'm
no longer seeing rate limiting behavior significantly different from the
norm.

I've noticed that many result pages are coming back with empty results but
if I re-request the same page (after a couple second delay), I can often get
results for that page. These are for queries with very low tweet velocity,
so it's not like these are new results coming in. Is this related to
http://status.twitter.com/post/3785043723/slow-searches ?

Thanks.

Hayes

On Mon, Mar 21, 2011 at 6:13 PM, Matt Harris wrote:

> In many cases we are forced to change the rate limits in response to
> a significant increase in requests, which means it isn't always possible to
> give advanced notice of rate limit changes.
>
> For some of you it sounds like your code that handles rate limiting didn't
> react appropriately. When receiving a 420 response we recommend you stop
> making requests and then after the retry-after, slowly build up the number
> of requests you make. Put another way it isn't a good idea to make requests
> to the API at the velocity that caused the 420 response before.
>
> As always, the rate limits are there to ensure the system is responsive and
> available to as many users as possible. This means it is necessary to reduce
> the number of queries you can make without notice.
>
> The best place to stay informed about issues like this are posted through
> @twitterapi and published on the Twitter status blog:
> http://status.twitter.com/post/3785043723/slow-searches
>
> Best,
> @themattharris
> Developer Advocate, Twitter
> http://twitter.com/themattharris
>
>
>
> On Mon, Mar 21, 2011 at 7:41 AM, Colin Surprenant <
> colin.surpren...@gmail.com> wrote:
>
>> By adjusting the rate limits to reduce the stress on your search api
>> without notice you have significantly increased the stress level on
>> our end :P Seriously, advanced notice of the situation would have been
>> welcome.
>>
>> In particular what created lots of confusion on our end is that even
>> after pausing for the specified "retry_after" delay we would
>> immediately get repeated 420s at which point we started to assume our
>> IPs were banned (which also contributed to increase the stress level).
>>
>> Colin
>>
>> On Mar 21, 9:12 am, Jeffrey Greenberg 
>> wrote:
>> > Taylor,
>> > Yeah this was definitely NOT good.In the past, when there is a
>> > service disruption, your api group would post something on your status
>> > page and tweet about it... Instead, I'm finding out about this from my
>> > customers...
>> >
>> > Did y'all tweet about this or present this somewhere where I could find
>> it?
>> >
>> > Jeffrey
>> > Tweettronics.com
>> >
>> > On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >  wrote:
>> > > Without prior notice, I can understand (circumstances), but without
>> > > any kind of subsequent announcement?? Means we have to discover issues
>> > > ourselves, verify that they're Twitter related (and not internal),
>> > > then search around for existing discussion on the topic. Saves us a
>> > > lot of time and headaches if Twitter would just announce stuff like
>> > > this.
>> >
>> > > On Mar 18, 2:51 pm, Taylor Singletary 
>> > > wrote:
>> > >> We're working to reinstate the usual limits on the Search API; due to
>> the
>> > >> impact of the Japanese earthquake and resultant query increase
>> against the
>> > >> Search API, some rates were adjusted to cope & better serve queries.
>> Will
>> > >> give everyone an update with the various limits are adjusted.
>> >
>> > >> @episod  - Taylor Singletary - Twitter
>> Developer
>> > >> Advocate
>> >
>> > >> On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis 
>> wrote:
>> > >> > Hi,
>> >
>> > >> > We're seeing this as well starting at approximately the same time
>> as
>> > >> > described. We've backed off on searching but are seeing no
>> reduction in the
>> > >> > sporadic limiting. It also appears that the amount of results
>> returned on
>> > >> > successful queries is severely limited. Some queries that often
>> have 1500
>> > >> > tweets from the last 5 days are returning far fewer results from
>> only the
>> > >> > last day.
>> >
>> > >> > Could we get an update on this?
>> >
>> > >> > Hayes
>> >
>> > >> > On Fri, Mar 18, 2011 at 10:13 AM, Eric  wrote:
>> >
>> > >> >> We're also seeing 400s on different boxes across different IP
>> > >> >> addresses with different queries (so it does not appear to be
>> server
>> > >> >> or query specific). These began on all boxes at 2 a.m. UTC. We've
>> > >> >> backed off on both number and rate of queries with no effect.
>> We've
>> > >> >> also noticed an increase in sporadic fail whales via browser based
>> > >> >> search (atom and html) from personal accounts, although we haven't
>> > >> >> attempted to quantify it.
>> >
>> > >> >> On Mar 18, 7:40 am, zaver  wrote:
>> > >> >> > Hello,
>> >
>> > >> >> > After the lat

Re: [twitter-dev] Re: Search API rate limit change?

[Matt Harris]

In many cases we are forced to change the rate limits in response to
a significant increase in requests, which means it isn't always possible to
give advanced notice of rate limit changes.

For some of you it sounds like your code that handles rate limiting didn't
react appropriately. When receiving a 420 response we recommend you stop
making requests and then after the retry-after, slowly build up the number
of requests you make. Put another way it isn't a good idea to make requests
to the API at the velocity that caused the 420 response before.

As always, the rate limits are there to ensure the system is responsive and
available to as many users as possible. This means it is necessary to reduce
the number of queries you can make without notice.

The best place to stay informed about issues like this are posted through
@twitterapi and published on the Twitter status blog:
http://status.twitter.com/post/3785043723/slow-searches

Best,
@themattharris
Developer Advocate, Twitter
http://twitter.com/themattharris


On Mon, Mar 21, 2011 at 7:41 AM, Colin Surprenant <
colin.surpren...@gmail.com> wrote:

> By adjusting the rate limits to reduce the stress on your search api
> without notice you have significantly increased the stress level on
> our end :P Seriously, advanced notice of the situation would have been
> welcome.
>
> In particular what created lots of confusion on our end is that even
> after pausing for the specified "retry_after" delay we would
> immediately get repeated 420s at which point we started to assume our
> IPs were banned (which also contributed to increase the stress level).
>
> Colin
>
> On Mar 21, 9:12 am, Jeffrey Greenberg 
> wrote:
> > Taylor,
> > Yeah this was definitely NOT good.In the past, when there is a
> > service disruption, your api group would post something on your status
> > page and tweet about it... Instead, I'm finding out about this from my
> > customers...
> >
> > Did y'all tweet about this or present this somewhere where I could find
> it?
> >
> > Jeffrey
> > Tweettronics.com
> >
> > On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner
> >
> >
> >
> >
> >
> >
> >
> >  wrote:
> > > Without prior notice, I can understand (circumstances), but without
> > > any kind of subsequent announcement?? Means we have to discover issues
> > > ourselves, verify that they're Twitter related (and not internal),
> > > then search around for existing discussion on the topic. Saves us a
> > > lot of time and headaches if Twitter would just announce stuff like
> > > this.
> >
> > > On Mar 18, 2:51 pm, Taylor Singletary 
> > > wrote:
> > >> We're working to reinstate the usual limits on the Search API; due to
> the
> > >> impact of the Japanese earthquake and resultant query increase against
> the
> > >> Search API, some rates were adjusted to cope & better serve queries.
> Will
> > >> give everyone an update with the various limits are adjusted.
> >
> > >> @episod  - Taylor Singletary - Twitter
> Developer
> > >> Advocate
> >
> > >> On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis 
> wrote:
> > >> > Hi,
> >
> > >> > We're seeing this as well starting at approximately the same time as
> > >> > described. We've backed off on searching but are seeing no reduction
> in the
> > >> > sporadic limiting. It also appears that the amount of results
> returned on
> > >> > successful queries is severely limited. Some queries that often have
> 1500
> > >> > tweets from the last 5 days are returning far fewer results from
> only the
> > >> > last day.
> >
> > >> > Could we get an update on this?
> >
> > >> > Hayes
> >
> > >> > On Fri, Mar 18, 2011 at 10:13 AM, Eric  wrote:
> >
> > >> >> We're also seeing 400s on different boxes across different IP
> > >> >> addresses with different queries (so it does not appear to be
> server
> > >> >> or query specific). These began on all boxes at 2 a.m. UTC. We've
> > >> >> backed off on both number and rate of queries with no effect. We've
> > >> >> also noticed an increase in sporadic fail whales via browser based
> > >> >> search (atom and html) from personal accounts, although we haven't
> > >> >> attempted to quantify it.
> >
> > >> >> On Mar 18, 7:40 am, zaver  wrote:
> > >> >> > Hello,
> >
> > >> >> > After the latest performance issues with the search api i have
> been
> > >> >> > seeing a lot of 420 response codes.From yesterday until now i
> only get
> > >> >> > 420 responses on the every search i make. In particular, i search
> for
> > >> >> > about 100 keywords simultaneously every 6 mins. Why is this
> happening?
> > >> >> > Was there any change on the Search API limit?
> >
> > >> >> > Any help is greatly appreciated.
> >
> > >> >> > Thanks,
> > >> >> > Zaver
> >
> > >> >> --
> > >> >> Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> > >> >> API updates via Twitter:http://twitter.com/twitterapi
> > >> >> Issues/Enhancements Tracker:
> > >> >>http://code.google.com/p/twitter-api/issues/list
> 

Re: [twitter-dev] Re: Search API rate limit change?

[Jeffrey Greenberg]

Taylor,
Yeah this was definitely NOT good.In the past, when there is a
service disruption, your api group would post something on your status
page and tweet about it... Instead, I'm finding out about this from my
customers...

Did y'all tweet about this or present this somewhere where I could find it?

Jeffrey
Tweettronics.com

On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner
 wrote:
> Without prior notice, I can understand (circumstances), but without
> any kind of subsequent announcement?? Means we have to discover issues
> ourselves, verify that they're Twitter related (and not internal),
> then search around for existing discussion on the topic. Saves us a
> lot of time and headaches if Twitter would just announce stuff like
> this.
>
> On Mar 18, 2:51 pm, Taylor Singletary 
> wrote:
>> We're working to reinstate the usual limits on the Search API; due to the
>> impact of the Japanese earthquake and resultant query increase against the
>> Search API, some rates were adjusted to cope & better serve queries. Will
>> give everyone an update with the various limits are adjusted.
>>
>> @episod  - Taylor Singletary - Twitter Developer
>> Advocate
>>
>> On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis  wrote:
>> > Hi,
>>
>> > We're seeing this as well starting at approximately the same time as
>> > described. We've backed off on searching but are seeing no reduction in the
>> > sporadic limiting. It also appears that the amount of results returned on
>> > successful queries is severely limited. Some queries that often have 1500
>> > tweets from the last 5 days are returning far fewer results from only the
>> > last day.
>>
>> > Could we get an update on this?
>>
>> > Hayes
>>
>> > On Fri, Mar 18, 2011 at 10:13 AM, Eric  wrote:
>>
>> >> We're also seeing 400s on different boxes across different IP
>> >> addresses with different queries (so it does not appear to be server
>> >> or query specific). These began on all boxes at 2 a.m. UTC. We've
>> >> backed off on both number and rate of queries with no effect. We've
>> >> also noticed an increase in sporadic fail whales via browser based
>> >> search (atom and html) from personal accounts, although we haven't
>> >> attempted to quantify it.
>>
>> >> On Mar 18, 7:40 am, zaver  wrote:
>> >> > Hello,
>>
>> >> > After the latest performance issues with the search api i have been
>> >> > seeing a lot of 420 response codes.From yesterday until now i only get
>> >> > 420 responses on the every search i make. In particular, i search for
>> >> > about 100 keywords simultaneously every 6 mins. Why is this happening?
>> >> > Was there any change on the Search API limit?
>>
>> >> > Any help is greatly appreciated.
>>
>> >> > Thanks,
>> >> > Zaver
>>
>> >> --
>> >> Twitter developer documentation and resources:http://dev.twitter.com/doc
>> >> API updates via Twitter:http://twitter.com/twitterapi
>> >> Issues/Enhancements Tracker:
>> >>http://code.google.com/p/twitter-api/issues/list
>> >> Change your membership to this group:
>> >>http://groups.google.com/group/twitter-development-talk
>>
>> >  --
>> > Twitter developer documentation and resources:http://dev.twitter.com/doc
>> > API updates via Twitter:http://twitter.com/twitterapi
>> > Issues/Enhancements Tracker:
>> >http://code.google.com/p/twitter-api/issues/list
>> > Change your membership to this group:
>> >http://groups.google.com/group/twitter-development-talk
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group: 
> http://groups.google.com/group/twitter-development-talk
>

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk

Re: [twitter-dev] Re: Search API rate limit change?

[Taylor Singletary]

We're working to reinstate the usual limits on the Search API; due to the
impact of the Japanese earthquake and resultant query increase against the
Search API, some rates were adjusted to cope & better serve queries. Will
give everyone an update with the various limits are adjusted.

@episod  - Taylor Singletary - Twitter Developer
Advocate


On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis  wrote:

> Hi,
>
> We're seeing this as well starting at approximately the same time as
> described. We've backed off on searching but are seeing no reduction in the
> sporadic limiting. It also appears that the amount of results returned on
> successful queries is severely limited. Some queries that often have 1500
> tweets from the last 5 days are returning far fewer results from only the
> last day.
>
> Could we get an update on this?
>
> Hayes
>
>
>
> On Fri, Mar 18, 2011 at 10:13 AM, Eric  wrote:
>
>> We're also seeing 400s on different boxes across different IP
>> addresses with different queries (so it does not appear to be server
>> or query specific). These began on all boxes at 2 a.m. UTC. We've
>> backed off on both number and rate of queries with no effect. We've
>> also noticed an increase in sporadic fail whales via browser based
>> search (atom and html) from personal accounts, although we haven't
>> attempted to quantify it.
>>
>> On Mar 18, 7:40 am, zaver  wrote:
>> > Hello,
>> >
>> > After the latest performance issues with the search api i have been
>> > seeing a lot of 420 response codes.From yesterday until now i only get
>> > 420 responses on the every search i make. In particular, i search for
>> > about 100 keywords simultaneously every 6 mins. Why is this happening?
>> > Was there any change on the Search API limit?
>> >
>> > Any help is greatly appreciated.
>> >
>> > Thanks,
>> > Zaver
>>
>> --
>> Twitter developer documentation and resources: http://dev.twitter.com/doc
>> API updates via Twitter: http://twitter.com/twitterapi
>> Issues/Enhancements Tracker:
>> http://code.google.com/p/twitter-api/issues/list
>> Change your membership to this group:
>> http://groups.google.com/group/twitter-development-talk
>>
>
>  --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk

Re: [twitter-dev] Re: Search API rate limit change?

[Hayes Davis]

Hi,

We're seeing this as well starting at approximately the same time as
described. We've backed off on searching but are seeing no reduction in the
sporadic limiting. It also appears that the amount of results returned on
successful queries is severely limited. Some queries that often have 1500
tweets from the last 5 days are returning far fewer results from only the
last day.

Could we get an update on this?

Hayes


On Fri, Mar 18, 2011 at 10:13 AM, Eric  wrote:

> We're also seeing 400s on different boxes across different IP
> addresses with different queries (so it does not appear to be server
> or query specific). These began on all boxes at 2 a.m. UTC. We've
> backed off on both number and rate of queries with no effect. We've
> also noticed an increase in sporadic fail whales via browser based
> search (atom and html) from personal accounts, although we haven't
> attempted to quantify it.
>
> On Mar 18, 7:40 am, zaver  wrote:
> > Hello,
> >
> > After the latest performance issues with the search api i have been
> > seeing a lot of 420 response codes.From yesterday until now i only get
> > 420 responses on the every search i make. In particular, i search for
> > about 100 keywords simultaneously every 6 mins. Why is this happening?
> > Was there any change on the Search API limit?
> >
> > Any help is greatly appreciated.
> >
> > Thanks,
> > Zaver
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>

-- 
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group: 
http://groups.google.com/group/twitter-development-talk