Large companies, and anyone who wants sustained access to high-volume Twitter data, use the Streaming API. The REST API is primarily for client applications and very-small-scale integrations. Note that REST whitelisting is unlikely for use-cases that are covered by the Streaming API.
-John Kalucki http://twitter.com/jkalucki Twitter, Inc. On Mon, Sep 20, 2010 at 8:44 AM, Vijay <vijayaraghura...@gmail.com> wrote: > OK Tom, will queue it and spread it, that should give me more room. > > But that still doesn't answer the question - how do the big companies > manage? > Do they have multiple IPs? > > On Mon, Sep 20, 2010 at 11:41 AM, Tom van der Woerdt <i...@tvdw.eu> wrote: > >> Just don't and you'll be fine :-) Just queue the requests and spread >> them over a few hours. It may also help to setup a Twitter account, as >> it will allow you to make 350 requests per hour instead. >> >> Tom >> >> >> On 9/20/10 5:38 PM, Vijay wrote: >> > >> > "You really shouldn't be planning for the case where every single >> Twitter >> > user uses your application" >> > >> > Haha, I agree. Its just that I took a few thousand users randomly, and >> > was playing >> > with their publicly available stats, tweets etc and quickly hit the rate >> > limit. Hence >> > this question. >> > >> > On Mon, Sep 20, 2010 at 11:21 AM, Tom van der Woerdt <i...@tvdw.eu >> > <mailto:i...@tvdw.eu>> wrote: >> > >> > You really shouldn't be planning for the case where every single >> Twitter >> > user uses your application ;-) When you get to 100 users, you >> request >> > whitelisting. When you get to 20000 users, your server will have a >> > problem handling all the information and you'll need to get a second >> > server anyway. >> > >> > Tom >> > >> > >> > On 9/20/10 5:17 PM, Vijay wrote: >> > > I am not sure about 20k, but 150 is miniscule. >> > > If I am collecting stats on a bunch of users every hour for >> example, I >> > > can only >> > > collect on 150 users, which is tiny, compared to 140 million users >> > > Twitter has. >> > > >> > > How do the big companies manage? For example, twittercounter >> > claims to >> > > have stats on 10 million users...that is a LOT of data. >> > > >> > > On Mon, Sep 20, 2010 at 11:14 AM, Tom van der Woerdt < >> i...@tvdw.eu >> > <mailto:i...@tvdw.eu> >> > > <mailto:i...@tvdw.eu <mailto:i...@tvdw.eu>>> wrote: >> > > >> > > 20k per IP is a lot. It means that a server should make more >> > than 5 >> > > requests per second to hit the limit, which is a lot. >> > > >> > > About your options: no idea. Just make sure to use the proper >> > functions >> > > and try not to hit the limits? :-) 150 is a lot as well, most >> > Desktop >> > > clients don't hit it (didn't, until Lists came ^^). >> > > >> > > Tom >> > > >> > > >> > > On 9/20/10 4:56 PM, Vijay wrote: >> > > > Hi, >> > > > >> > > > Newbie question, so please bear with me. >> > > > >> > > > I am experimenting with twitter API, but quickly found >> > myself hitting >> > > > the rate limit >> > > > (150 without authentication, correct?) >> > > > >> > > > How do the big sites get over the rate limit (tweetstats, >> > > twittercounter >> > > > etc)? >> > > > Twitter's documentation says they can white list IPs, but >> > would still >> > > > allow only >> > > > 20k requests per hour. Also, they would only white list >> > apps that are >> > > > already in >> > > > production. So if I just want to experiment / learn, this >> > > wouldn't work. >> > > > >> > > > What are my options? There is no way I can have multiple >> IPs. >> > > > >> > > > Vijay. >> > > > >> > > > -- >> > > > Twitter developer documentation and resources: >> > > http://dev.twitter.com/doc >> > > > API updates via Twitter: http://twitter.com/twitterapi >> > > > Issues/Enhancements Tracker: >> > > > http://code.google.com/p/twitter-api/issues/list >> > > > Change your membership to this group: >> > > > >> http://groups.google.com/group/twitter-development-talk?hl=en >> > > >> > > -- >> > > Twitter developer documentation and resources: >> > > http://dev.twitter.com/doc >> > > API updates via Twitter: http://twitter.com/twitterapi >> > > Issues/Enhancements Tracker: >> > > http://code.google.com/p/twitter-api/issues/list >> > > Change your membership to this group: >> > > http://groups.google.com/group/twitter-development-talk?hl=en >> > > >> > > >> > > -- >> > > Twitter developer documentation and resources: >> > http://dev.twitter.com/doc >> > > API updates via Twitter: http://twitter.com/twitterapi >> > > Issues/Enhancements Tracker: >> > > http://code.google.com/p/twitter-api/issues/list >> > > Change your membership to this group: >> > > http://groups.google.com/group/twitter-development-talk?hl=en >> > >> > -- >> > Twitter developer documentation and resources: >> > http://dev.twitter.com/doc >> > API updates via Twitter: http://twitter.com/twitterapi >> > Issues/Enhancements Tracker: >> > http://code.google.com/p/twitter-api/issues/list >> > Change your membership to this group: >> > http://groups.google.com/group/twitter-development-talk?hl=en >> > >> > >> > -- >> > Twitter developer documentation and resources: >> http://dev.twitter.com/doc >> > API updates via Twitter: http://twitter.com/twitterapi >> > Issues/Enhancements Tracker: >> > http://code.google.com/p/twitter-api/issues/list >> > Change your membership to this group: >> > http://groups.google.com/group/twitter-development-talk?hl=en >> >> -- >> Twitter developer documentation and resources: http://dev.twitter.com/doc >> API updates via Twitter: http://twitter.com/twitterapi >> Issues/Enhancements Tracker: >> http://code.google.com/p/twitter-api/issues/list >> Change your membership to this group: >> http://groups.google.com/group/twitter-development-talk?hl=en >> > > -- > Twitter developer documentation and resources: http://dev.twitter.com/doc > API updates via Twitter: http://twitter.com/twitterapi > Issues/Enhancements Tracker: > http://code.google.com/p/twitter-api/issues/list > Change your membership to this group: > http://groups.google.com/group/twitter-development-talk?hl=en > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
