[twitter-dev] Search API Rate Limiting
I'm developing an application that uses the Twitter Search API. The app searches Twitter for many (at least a couple thousand) specific keywords in real time. A server would be set up to get the results for the many keywords in tweets and store them in a database that the application would access to return the results, as opposed to every user searching for the keywords on their end every time. The problem is after a while I notice that Twitter stops responding with new results. Is Twitter caching the results after a set amount of requests I can make in a given time period? The FAQ section on the developer's wiki does not provide much info on the rate limiting of the Search API other than that it is generous. Do you guys have the inside scoop on the actual numbers? I see where to submit an application and the application is basically working, but my client hasn't finalized the details for the app like the name and url yet. Am I out of luck until I am ready to officially submit the application for whitelisting. If not, how can I get my application's server added to a whitelist to ensure that the tweets return from twitter are real-time? -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk
Re: [twitter-dev] Search API Rate Limiting
Hi, sounds like you really want to look into the streaming API instead. http://dev.twitter.com/pages/streaming_api_methods#statuses-filter cheers -m On Thu, Jun 2, 2011 at 1:00 PM, HRyba bhrdes...@gmail.com wrote: I'm developing an application that uses the Twitter Search API. The app searches Twitter for many (at least a couple thousand) specific keywords in real time. A server would be set up to get the results for the many keywords in tweets and store them in a database that the application would access to return the results, as opposed to every user searching for the keywords on their end every time. The problem is after a while I notice that Twitter stops responding with new results. Is Twitter caching the results after a set amount of requests I can make in a given time period? The FAQ section on the developer's wiki does not provide much info on the rate limiting of the Search API other than that it is generous. Do you guys have the inside scoop on the actual numbers? I see where to submit an application and the application is basically working, but my client hasn't finalized the details for the app like the name and url yet. Am I out of luck until I am ready to officially submit the application for whitelisting. If not, how can I get my application's server added to a whitelist to ensure that the tweets return from twitter are real-time? -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Will someone from Twitter please respond if there is an ETA to resolve this issue. Work arounds can never be really as effective as the real deal.
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Chad, Sorry for not being clear. I was thinking about Abraham William's suggestion above where Twitter Search API works with authenticated sessions+rate limiting, instead of IP based rate filtering. Just so you know, AppEngine has 30 second timeout on request to all AppEngine urls, and 10 second timeout on each individual HTTP request made within an AppEngine request. In case you are making multiple HTTP requests to Twitter within each individual AppEngine request, all the communication microseconds, from AppEngine to Proxy and Proxy to Twitter and then Twitter to Proxy and Proxy to AppEngine, quickly addup leading to timeouts. Personally i have tried quite a few scenarios to catch all the data i can, but from my experience, i can catch only 30%(sometimes better, sometimes almost nothing) of what i want, and rest just ends up with 503 and eventually since_id/max_id getting too old to get response from the Twitter Search API. So, right now Twitter is putting it's resources to offer a very robust Search API, but we as developers cannot use it effectively just 'cause of the way the hits are counted. Not to mention we are also investing funds to keep our apps running. Hope you understand our position. Thanks On Oct 18, 3:12 pm, Chad Etzel c...@twitter.com wrote: On Sun, Oct 18, 2009 at 8:09 AM, vivekpuri v...@vivekpuri.com wrote: Will someone from Twitter please respond if there is an ETA to resolve this issue. Work arounds can never be really as effective as the real deal. Sorry, I thought it was clear from the previous email. There is no ETA because it's not going to be resolved. GAE does not use an IP infrastructure that is amicable to our rate-limiting logic, so if you want to integrate IP rate-limited calls into your web-based applications, you will need to either use the workaround stated earlier or use a hosting service that will let you use a static IP. -Chad
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I would recommend just using a physical server and uploading a simple php proxy script. If you have existing webspace, it will save you the trouble of setting up an complete ec2 build just to run a proxy script. On Oct 9, 7:11 pm, Akshar akshar.d...@gmail.com wrote: Thanks Abraham. Any pointers on how to setup a proxy on amazon ec2 for GAE? On Oct 8, 6:07 pm, Abraham Williams 4bra...@gmail.com wrote: Pretty much. You have limited options: 1) Run your Search API requests through a proxy where you will have exclusive access to the IP. 2) Wait for V2 of the Twitter API where the REST and Search APIs get combined so you can have authenticated search queries. 3) Hope Twitter slaps some duct tape on the issue and rolls out a whitelisting method for the Search API that includes passkeys in your user agent or some such thing. 4) Develop on non cloud base infrastructure. 5) Something else. Abraham 2009/10/8 Akshar akshar.d...@gmail.com http://apiwiki.twitter.com/Rate-limitingstatesthat for cloud platforms like Google App Engine, applications without a static IP addresses cannot receive Search whitelisting. Does that mean there is no way to avoid getting HTTP 503 response codes to search requests from app engine? On Oct 8, 2:09 pm, Akshar akshar.d...@gmail.com wrote: Any other solutions available for app engine folks stuck out here? Please help! I'm noticing this exact problem as well. I'm making only a few requests per hour. I have tried setting the user-agent but it did not help. Akshar On Oct 6, 9:50 am, Chad Etzel c...@twitter.com wrote: Hi All, GAE sites are problematic for the Twitter/SearchAPIbecause the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the TwitterSearchAPIget the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access theirsearchAPI. It seems like they have really started blocking searchrequests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using thesearchAPIand it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abrahamhttp://web608.org/geeks/abraham/blogs/2009/... This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Thanks Abraham. Any pointers on how to setup a proxy on amazon ec2 for GAE? On Oct 8, 6:07 pm, Abraham Williams 4bra...@gmail.com wrote: Pretty much. You have limited options: 1) Run your Search API requests through a proxy where you will have exclusive access to the IP. 2) Wait for V2 of the Twitter API where the REST and Search APIs get combined so you can have authenticated search queries. 3) Hope Twitter slaps some duct tape on the issue and rolls out a whitelisting method for the Search API that includes passkeys in your user agent or some such thing. 4) Develop on non cloud base infrastructure. 5) Something else. Abraham 2009/10/8 Akshar akshar.d...@gmail.com http://apiwiki.twitter.com/Rate-limitingstates that for cloud platforms like Google App Engine, applications without a static IP addresses cannot receive Search whitelisting. Does that mean there is no way to avoid getting HTTP 503 response codes to search requests from app engine? On Oct 8, 2:09 pm, Akshar akshar.d...@gmail.com wrote: Any other solutions available for app engine folks stuck out here? Please help! I'm noticing this exact problem as well. I'm making only a few requests per hour. I have tried setting the user-agent but it did not help. Akshar On Oct 6, 9:50 am, Chad Etzel c...@twitter.com wrote: Hi All, GAE sites are problematic for the Twitter/SearchAPIbecause the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the TwitterSearchAPIget the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access theirsearchAPI. It seems like they have really started blocking searchrequests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using thesearchAPIand it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abrahamhttp://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I have solved a problem like that: While I receive an error 503 - my application continue knocking to twitter with query. Everything works ;)
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
http://apiwiki.twitter.com/Rate-limiting states that for cloud platforms like Google App Engine, applications without a static IP addresses cannot receive Search whitelisting. Does that mean there is no way to avoid getting HTTP 503 response codes to search requests from app engine? On Oct 8, 2:09 pm, Akshar akshar.d...@gmail.com wrote: Any other solutions available for app engine folks stuck out here? Please help! I'm noticing this exact problem as well. I'm making only a few requests per hour. I have tried setting the user-agent but it did not help. Akshar On Oct 6, 9:50 am, Chad Etzel c...@twitter.com wrote: Hi All, GAE sites are problematic for the Twitter/SearchAPIbecause the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the TwitterSearchAPIget the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access theirsearchAPI. It seems like they have really started blocking searchrequests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using thesearchAPIand it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Pretty much. You have limited options: 1) Run your Search API requests through a proxy where you will have exclusive access to the IP. 2) Wait for V2 of the Twitter API where the REST and Search APIs get combined so you can have authenticated search queries. 3) Hope Twitter slaps some duct tape on the issue and rolls out a whitelisting method for the Search API that includes passkeys in your user agent or some such thing. 4) Develop on non cloud base infrastructure. 5) Something else. Abraham 2009/10/8 Akshar akshar.d...@gmail.com http://apiwiki.twitter.com/Rate-limiting states that for cloud platforms like Google App Engine, applications without a static IP addresses cannot receive Search whitelisting. Does that mean there is no way to avoid getting HTTP 503 response codes to search requests from app engine? On Oct 8, 2:09 pm, Akshar akshar.d...@gmail.com wrote: Any other solutions available for app engine folks stuck out here? Please help! I'm noticing this exact problem as well. I'm making only a few requests per hour. I have tried setting the user-agent but it did not help. Akshar On Oct 6, 9:50 am, Chad Etzel c...@twitter.com wrote: Hi All, GAE sites are problematic for the Twitter/SearchAPIbecause the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the TwitterSearchAPIget the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access theirsearchAPI. It seems like they have really started blocking searchrequests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using thesearchAPIand it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul -- Abraham Williams | Community Evangelist | http://web608.org Hacker | http://abrah.am | http://twitter.com/abraham http://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I am also facing this issue. I'm only making a couple of requests from GAE (about 3-4) and none of them are getting through, I keep getting the following using Twitter4J Twitter Exception while retrieving status twitter4j.TwitterException: 400:The request was invalid. An accompanying error message will explain why. This is the status code will be returned during rate limiting. ?xml version=1.0 encoding=UTF-8? hash request/statuses/show/2.xml/request errorRate limit exceeded. Clients may not make more than 150 requests per hour./error /hash On Oct 6, 7:13 pm, Paul Kinlan paul.kin...@gmail.com wrote: Hi Chad, I am sorry but that doesn't even help in the slightest. You are essentially saying that we shouldn't develop on the App Engine, since would now have to also buy a proxy. Which is completely unfeasible and defeats the purpose of why people are using the app engine. I understand that this might also be an App Engine issue - for instance they could have reduced the number of IP addresses they pool from to make external requests. This is a very noticeable change inratelimiting in the last few weeks. For instance I could run roughly 2 searches a second, then all of a sudden I would be lucky to run 2 every 15 seconds. User-Agent strings were supposed to allievate this issue. There are more than enough pieces of meta data on an App Engine request that Identify the exact application that is making the requests - I guess it is too much effort to take these into account. I am in the fortunate position that allowed me to set up a nginx proxy quickly, but I suspect a lot of other people couldn't do that. I hope something can be sorted for the large number of GAE based Twitter apps. Paul Kinlan On 6 Oct 2009, at 17:50, Chad Etzel c...@twitter.com wrote: Hi All, GAE sites are problematic for the Twitter/Search API because the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the Twitter Search API get the 503 error message indicatingrate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user- agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503ratelimit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get ratelimit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavilyratelimited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limitbeing hit and then all of a sudden only about 60% of the searches I perform will beratelimited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Twitter should really in this case either white list all GAE IPs (I'm sure an email to Google could get all IPs they use) or allow charging API requests to an authenticated account rather than by IP (much like the REST API does). This way each GAE application would just set up a twitter account and each gets its own 150 request per hour. Josh
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Same here; my app runs on Google App Engine and 40% of the requests to the Twitter Search API get the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi All, GAE sites are problematic for the Twitter/Search API because the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the Twitter Search API get the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi Chad, I am sorry but that doesn't even help in the slightest. You are essentially saying that we shouldn't develop on the App Engine, since would now have to also buy a proxy. Which is completely unfeasible and defeats the purpose of why people are using the app engine. I understand that this might also be an App Engine issue - for instance they could have reduced the number of IP addresses they pool from to make external requests. This is a very noticeable change in rate limiting in the last few weeks. For instance I could run roughly 2 searches a second, then all of a sudden I would be lucky to run 2 every 15 seconds. User-Agent strings were supposed to allievate this issue. There are more than enough pieces of meta data on an App Engine request that Identify the exact application that is making the requests - I guess it is too much effort to take these into account. I am in the fortunate position that allowed me to set up a nginx proxy quickly, but I suspect a lot of other people couldn't do that. I hope something can be sorted for the large number of GAE based Twitter apps. Paul Kinlan On 6 Oct 2009, at 17:50, Chad Etzel c...@twitter.com wrote: Hi All, GAE sites are problematic for the Twitter/Search API because the IPs making outgoing requests are fluid and cannot as such be easily allowed for access. Also, since most IPs are shared, other applications on the same IPs making requests mean that fewer requests per app get through. One work around would be to spin up a server in EC2 or Rackspace Cloud or something and use it as a proxy for your requests. That way you have a dedicated IP that will have its full share of resources talking with the Twitter servers. HTH, -Chad On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander moman...@google.com wrote: Same here; my app runs on Google App Engine and 40% of the requests to the Twitter Search API get the 503 error message indicating rate limiting. Is there anything we as app authors can do on our side to alleviate the problem? /Martin On Oct 5, 1:53 pm, Paul Kinlan paul.kin...@gmail.com wrote: I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user- agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I'm noticing this problem as well. I'm making only a couple requests per hour. I have tried setting the user-agent and the HTTP_REFERER headers to a custom name, but Twitter doesn't seem to care. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate limiting - App Engine (again)
I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request. 2009/10/5 elkelk danielshaneup...@gmail.com Hi all, I am having the same issue. I have tried setting a custom user-agent, but this doesn't seem to affect the fact that twitter is limiting based on I.P. address. I'm only making about 5 searches an hour and 80% of them are failing on app engine due to a 503 rate limit. Twitter needs to determine a better way to let cloud clients access their search API. It seems like they have really started blocking search requests in the last week or so. If anyone has any idea about how to better identify my app engine app please let let me know. On Oct 5, 2:59 am, steel steel...@gmail.com wrote: Hi. I have this problem too. My application does two request per hour and it get rate limit. What is wrong? I think it is twitter's problems On 1 окт, 01:45, Paul Kinlan paul.kin...@gmail.com wrote: Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Search API Rate limiting - App Engine (again)
Hi Guys, I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days. I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so. Has something changed recently? Paul
[twitter-dev] Re: Search API Rate Limiting and Performance
Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
The Search team is working on indexing latency and throughput, along with a many other things. There have been big improvements recently and more are on the way. In the mean time, if you need closer to real-time results, consider the track parameter on the Streaming API. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote: Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
John, the original message of this thread is about rate limit being totally erratic, as several users have noticed. here is the detail of what I'm seeing: http://groups.google.com/group/twitter-development-talk/browse_thread/thread/40c82b4dbc0536bd Here is another user reporting the problem : http://groups.google.com/group/twitter-development-talk/browse_thread/thread/51bd6667d3b0a0a6 Basically the X-RateLimit-Remaining header goes up and down and up again between adjacent requests (all authenticated requests). The result is that the limit can appear to be reached way before 150 requests were issued... Thanks On Sep 4, 6:30 pm, John Kalucki jkalu...@gmail.com wrote: The Search team is working on indexing latency and throughput, along with a many other things. There have been big improvements recently and more are on the way. In the mean time, if you need closer to real-time results, consider the track parameter on the Streaming API. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote: Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm,Dewald Pretoriusdpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
Various APIs have their own rate limiting mechanisms. The www, search and streaming rate limits are all customized to their usage patterns and share little to no code and/or state. -John On Sep 4, 9:49 am, Reivax xavier.yo...@gmail.com wrote: John, the original message of this thread is about rate limit being totally erratic, as several users have noticed. here is the detail of what I'm seeing: http://groups.google.com/group/twitter-development-talk/browse_thread... Here is another user reporting the problem :http://groups.google.com/group/twitter-development-talk/browse_thread... Basically the X-RateLimit-Remaining header goes up and down and up again between adjacent requests (all authenticated requests). The result is that the limit can appear to be reached way before 150 requests were issued... Thanks On Sep 4, 6:30 pm, John Kalucki jkalu...@gmail.com wrote: The Search team is working on indexing latency and throughput, along with a many other things. There have been big improvements recently and more are on the way. In the mean time, if you need closer to real-time results, consider the track parameter on the Streaming API. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter Inc. On Sep 4, 6:58 am, pxpilot pxpi...@gmail.com wrote: Search API will rock if it would only be reliable what we see looks to be some sort of a funky cache, a query (atom) can be missing some latest tweets and then after a while they show up, if you tweak the query you can see 'em. you ever seen this problem? also what did you do special with user agent? On Sep 2, 12:12 pm, Amitab hiamita...@gmail.com wrote: I have exchanged emails with Twitter on this and I believe they are working on it. We use search extensively atwww.Twaller.com. The errors in search that we are seing is as follows: (1) HTTP status code: 403 Message:The request is understood, but it has been refused. An accompanying error message will explain why. (2)HTTP status code: 503 Message:Service Unavailable: The Twitter servers are up, but overloaded with requests. Try again later. The search and trend methods use this to indicate when you are being rate limited. (3) HTTP status code: -1 Message:Read timed out These errors are becoming more and more frequent lately. /Amitab On Sep 1, 5:41 pm,Dewald Pretoriusdpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
[twitter-dev] Re: Search API Rate Limiting and Performance
Dewald, I'm not on the search team, but there are a lot of discussions over there this morning about search api rate limits and related issues. Search rate limiting issues (vs. www.twitter.com or api.twitter.com) probably boil down to one of three categories: 1) Search service interruptions - We're working on these, obviously. You may see odd behavior when search capacity is reduced due to some operational issue, and the system is running far too hot. 2) General API limit reductions - We're slightly less generous than we have been in the past due to persistent abuse of the search api and due to overall capacity issues. We can't always provide for all valid use cases, although that is certainly a high priority. 3) Bugs. There could be defects that prevent valid use of the API. Detailed data of limiting mishaps would be helpful here. Timestamps, queries and statistics around reasonable use that was denied may allow us to identify a previously unknown defect. The search team is quite interested in working on #3. -John Kalucki http://twitter.com/jkalucki Services, Twitter Inc. On Sep 1, 5:41 pm, Dewald Pretorius dpr...@gmail.com wrote: Twitter team, can you please do something about the performance and rate limiting of the Search API. It is becoming completely unworkable. I have jumped through all the hoops, with unique User Agents, sleeping my scripts in-between API calls, and yet the rate limiting is just becoming more severe, and the performance is just becoming slower and slower. Please help. A lot of us are using Search to provide services, and I am sure it is not only my service that is hurting very much under the present circumstances. Dewald
Re: Search API Rate Limiting
Ah, gotcha! You can, it will just display a browser warning. Which is not what you want :P The Terms say: We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. Try emailing, Alex Payne, or someone at Twitter about a whitelist. On Dec 7, 3:36 pm, Chad Etzel [EMAIL PROTECTED] wrote: No, you can't do an ajax authenticated GET or POST to a 3rd-party site. I am dynamically loading the json in the clients' browser. I would rather know the rate limits so I can abide by them. -Chad On Sun, Dec 7, 2008 at 10:42 AM, fastest963 [EMAIL PROTECTED] wrote: Since your doing this via AJAX and such, this may not be a good idea, but you could try passing a login to Twitter and having that login whitelisted?
Re: Search API Rate Limiting
The Terms say: We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. ...yes, which is exactly why I am asking the question in the first place. My code already handles the error case so no browser warnings are popped. I addressed the question to Matt originally since I thought he was the Search API guru, or am I mistaken? -Chad On Mon, Dec 8, 2008 at 10:23 AM, fastest963 [EMAIL PROTECTED] wrote: Ah, gotcha! You can, it will just display a browser warning. Which is not what you want :P The Terms say: We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. Try emailing, Alex Payne, or someone at Twitter about a whitelist. On Dec 7, 3:36 pm, Chad Etzel [EMAIL PROTECTED] wrote: No, you can't do an ajax authenticated GET or POST to a 3rd-party site. I am dynamically loading the json in the clients' browser. I would rather know the rate limits so I can abide by them. -Chad On Sun, Dec 7, 2008 at 10:42 AM, fastest963 [EMAIL PROTECTED] wrote: Since your doing this via AJAX and such, this may not be a good idea, but you could try passing a login to Twitter and having that login whitelisted?
Re: Search API Rate Limiting
Matt is the Search API guru, indeed. On Mon, Dec 8, 2008 at 08:16, Chad Etzel [EMAIL PROTECTED] wrote: The Terms say: We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. ...yes, which is exactly why I am asking the question in the first place. My code already handles the error case so no browser warnings are popped. I addressed the question to Matt originally since I thought he was the Search API guru, or am I mistaken? -Chad On Mon, Dec 8, 2008 at 10:23 AM, fastest963 [EMAIL PROTECTED] wrote: Ah, gotcha! You can, it will just display a browser warning. Which is not what you want :P The Terms say: We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. Try emailing, Alex Payne, or someone at Twitter about a whitelist. On Dec 7, 3:36 pm, Chad Etzel [EMAIL PROTECTED] wrote: No, you can't do an ajax authenticated GET or POST to a 3rd-party site. I am dynamically loading the json in the clients' browser. I would rather know the rate limits so I can abide by them. -Chad On Sun, Dec 7, 2008 at 10:42 AM, fastest963 [EMAIL PROTECTED] wrote: Since your doing this via AJAX and such, this may not be a good idea, but you could try passing a login to Twitter and having that login whitelisted? -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
Re: Change: Search API Rate Limiting
The error code for search rate limiting will be changing from HTTP 503 to HTTP 401 in the very near future (today or tomorrow). For details, continue reading. Are you sure you want to use 401 for this? 401 would indicate authorization required. If you're asking for credentials, that would make sense, but if you're not, I would think the 503 is still the proper response irrespective of broken proxies. I don't see other codes that have that one's temporal semantics. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * [EMAIL PROTECTED] -- If you have integrity, nothing else matters. -- Alan Simpson ---
Re: Change: Search API Rate Limiting
Of course right after sending a lengthy public email I see something that could let us keep 503 and fix the proxy errors. I'm working with operations on that, and if it does not pan out I'll confer with Alex on 400 versus 401. Stay tuned. — Matt On Dec 8, 2008, at 09:46 AM, Alex Payne wrote: We use 400 for rate limiting on the REST API. Matt and I are discussing whether or not this might be the correct response. Thoughts? On Mon, Dec 8, 2008 at 09:17, Cameron Kaiser [EMAIL PROTECTED] wrote: The error code for search rate limiting will be changing from HTTP 503 to HTTP 401 in the very near future (today or tomorrow). For details, continue reading. Are you sure you want to use 401 for this? 401 would indicate authorization required. If you're asking for credentials, that would make sense, but if you're not, I would think the 503 is still the proper response irrespective of broken proxies. I don't see other codes that have that one's temporal semantics. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * [EMAIL PROTECTED] -- If you have integrity, nothing else matters. -- Alan Simpson --- -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
Re: Change: Search API Rate Limiting
The error code for search rate limiting will be changing from HTTP 503 to HTTP 401 in the very near future (today or tomorrow). For details, continue reading. Are you sure you want to use 401 for this? 401 would indicate authorization required. If you're asking for credentials, that would make sense, but if you're not, I would think the 503 is still the proper response irrespective of broken proxies. I don't see other codes that have that one's temporal semantics. We use 400 for rate limiting on the REST API. Matt and I are discussing whether or not this might be the correct response. Thoughts? I guess that would work there too IMHO. It's ill-defined but that's a bonus in this case :) -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * [EMAIL PROTECTED] -- Logic is the art of going wrong with confidence. -- George Bernard Shaw
Re: Change: Search API Rate Limiting
You could compromise and do a 400.5 O_o On Mon, Dec 8, 2008 at 11:51, Matt Sanford [EMAIL PROTECTED] wrote: Of course right after sending a lengthy public email I see something that could let us keep 503 and fix the proxy errors. I'm working with operations on that, and if it does not pan out I'll confer with Alex on 400 versus 401. Stay tuned. — Matt On Dec 8, 2008, at 09:46 AM, Alex Payne wrote: We use 400 for rate limiting on the REST API. Matt and I are discussing whether or not this might be the correct response. Thoughts? On Mon, Dec 8, 2008 at 09:17, Cameron Kaiser [EMAIL PROTECTED] wrote: The error code for search rate limiting will be changing from HTTP 503 to HTTP 401 in the very near future (today or tomorrow). For details, continue reading. Are you sure you want to use 401 for this? 401 would indicate authorization required. If you're asking for credentials, that would make sense, but if you're not, I would think the 503 is still the proper response irrespective of broken proxies. I don't see other codes that have that one's temporal semantics. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * [EMAIL PROTECTED] -- If you have integrity, nothing else matters. -- Alan Simpson --- -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x -- | Abraham Williams | Web Developer | http://abrah.am | Brazen Careerist | Pro Hacker | http://www.brazencareerist.com | PoseurTech LLC | Mashup Ambassador | http://poseurte.ch | Web608 | Community Evangelist | http://web608.org | This email is: [] blogable [x] ask first [] private
Re: Change: Search API Rate Limiting
I think using 400 is much easy to handle the responses than using 401. Because I can use same http client code and same error handling code for both search API and REST API. In my case, I wrote a error handler which alerts a dialog whenever it gets a 401 because search API wouldn't return 401. On Mon, Dec 8, 2008 at 9:56 AM, Abraham Williams [EMAIL PROTECTED] wrote: You could compromise and do a 400.5 O_o On Mon, Dec 8, 2008 at 11:51, Matt Sanford [EMAIL PROTECTED] wrote: Of course right after sending a lengthy public email I see something that could let us keep 503 and fix the proxy errors. I'm working with operations on that, and if it does not pan out I'll confer with Alex on 400 versus 401. Stay tuned. — Matt On Dec 8, 2008, at 09:46 AM, Alex Payne wrote: We use 400 for rate limiting on the REST API. Matt and I are discussing whether or not this might be the correct response. Thoughts? On Mon, Dec 8, 2008 at 09:17, Cameron Kaiser [EMAIL PROTECTED] wrote: The error code for search rate limiting will be changing from HTTP 503 to HTTP 401 in the very near future (today or tomorrow). For details, continue reading. Are you sure you want to use 401 for this? 401 would indicate authorization required. If you're asking for credentials, that would make sense, but if you're not, I would think the 503 is still the proper response irrespective of broken proxies. I don't see other codes that have that one's temporal semantics. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * [EMAIL PROTECTED] -- If you have integrity, nothing else matters. -- Alan Simpson --- -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x -- | Abraham Williams | Web Developer | http://abrah.am | Brazen Careerist | Pro Hacker | http://www.brazencareerist.com | PoseurTech LLC | Mashup Ambassador | http://poseurte.ch | Web608 | Community Evangelist | http://web608.org | This email is: [] blogable [x] ask first [] private
Re: Search API Rate Limiting
No, you can't do an ajax authenticated GET or POST to a 3rd-party site. I am dynamically loading the json in the clients' browser. I would rather know the rate limits so I can abide by them. -Chad On Sun, Dec 7, 2008 at 10:42 AM, fastest963 [EMAIL PROTECTED] wrote: Since your doing this via AJAX and such, this may not be a good idea, but you could try passing a login to Twitter and having that login whitelisted?
Search API Rate Limiting
Hi Matt, I am noticing I am getting rate-limited by the Search API more and more frequently. I just got limited with a Retry-After value of 800 (or about 13 minutes). I'm not sure how much more my calm can be enhanced in a 13 minute period, but this does not bode well for my search apps such as TweetGrid.com. Can you say what exactly qualifies as exceeding the rate limit for search? There are no hard numbers in the docs. Is it possible to whitelist an app by referrer? TweetGrid works by querying the search servers from the clients' browsers, so whitelisting my server IP would be of no use. Request/Response headers are below: Thanks, -Chad Headers for reference: Request Headers Hostsearch.twitter.com User-AgentMozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1 Accept*/* Accept-Languageen-us,en;q=0.5 Accept-Encodinggzip,deflate Accept-CharsetISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive300 Connectionkeep-alive Referer http://tweetgrid.com/grid?l=1q1=jazzychadq2=tweetgrid+OR+%22tweet+grid%22 Cookierpp=100; __utmz=110314503.1227378583.81.2.utmcsr=twitter.com |utmccn=(referral)|utmcmd=referral|utmcct =/home; __utma=110314503.2390361217092253000.1216939739.1227380887.1228599709.83; __utma=43838368.64616265 .1208222991.1228589665.1228593169.206; __utmv=43838368.lang%3A%20en_US; __utmxx=43838368.2859631981918557 :1228059149:2592000; __utmx=43838368.2859631981918557:1:3; __utmz=43838368.1228108435.204.12.utmcsr =tweetgrid.com|utmccn=(referral)|utmcmd=referral|utmcct=/grid; __qca=1204080243-25856624-1326010; __utmc =43838368; _twitter_sess=BAh7CjoMY3NyZl9pZCIlMWFhZWZjZDY0ZjMxZjMyMWVhZjYxMjVhNjY4ZmY1%250AOGY6CXVzZXIwOg5yZXR1cm5fdG8iIWh0dHA6Ly90d2l0dGVyLmNvbS90d2Vl %250AdGdyaWQiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFz%250AaEhhc2h7AAY6CkB1c2VkewA6B2lkIiU5NGYzMTlkYmU5ZmI1M2U2MDg2OGNl %250ANjU5OTkzNjhmYQ%253D%253D--10fb9d01aceab6d2682b213515f18ae15dff5fe6; _search_twitter_sess=BAh7BzoMY3NyZl9pZCIlNjVkMjZlOGRjMjdiMWI0NWEwODZmMDg0ZjZmZGI3 %0AMTAiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh%0Ac2h7AAY6CkB1c2VkewA%3D--fc15a28d4723f2c07c2c77ef5dddb01e84a14843 ; __utmc=110314503 Response Headers DateSat, 06 Dec 2008 23:10:43 GMT Serverhi Status503 Service Unavailable Retry-After414 X-Runtime0.04179 Cache-Controlno-cache, max-age=300 Content-Typetext/html; charset=utf-8 X-Served-Bysearchweb006.twitter.com ExpiresSat, 06 Dec 2008 23:15:43 GMT VaryAccept-Encoding Content-Encodinggzip Content-Length66 Connectionclose
