Re: [twitter-dev] Search API Rate Limiting
Hi, sounds like you really want to look into the streaming API instead. http://dev.twitter.com/pages/streaming_api_methods#statuses-filter cheers -m On Thu, Jun 2, 2011 at 1:00 PM, HRyba bhrdes...@gmail.com wrote: I'm developing an application that uses the Twitter Search API. The app searches Twitter for many (at least a couple thousand) specific keywords in real time. A server would be set up to get the results for the many keywords in tweets and store them in a database that the application would access to return the results, as opposed to every user searching for the keywords on their end every time. The problem is after a while I notice that Twitter stops responding with new results. Is Twitter caching the results after a set amount of requests I can make in a given time period? The FAQ section on the developer's wiki does not provide much info on the rate limiting of the Search API other than that it is generous. Do you guys have the inside scoop on the actual numbers? I see where to submit an application and the application is basically working, but my client hasn't finalized the details for the app like the name and url yet. Am I out of luck until I am ready to officially submit the application for whitelisting. If not, how can I get my application's server added to a whitelist to ensure that the tweets return from twitter are real-time? -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk
Re: [twitter-dev] Search across multiple accounts
Hi Justin, you could follow (with the streaming api) users 1,2,and 3. But then you'd have to perform the search yourself. Alternatively, you can track (with the streaming api) your searchQuery and only keep tweets from users 1,2, and 3. Maybe I'm missing something here (e.g., userstreams?) -m On Wed, Feb 23, 2011 at 9:35 AM, Justin justin.realw...@gmail.com wrote: Is it possible to perform a search across multiple accounts? I've had a look though the search api and can't see anything but I could very well have missed something. What I'm thinking is something along the lines of: ?q=searchQuery+from%3Auser1,user2,user3 Also (again I can't see anything in the REST documentation straight away), is there a way to aggrigate streams of a defined list of users? Say; last X tweets from user1, user2 and user3? If not part of the twitter api, any suggestions? Many thanks Justin -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] URGENT: Advice on building the correct API stream
Hi, as far as I understand the docs there is no limit on the tweets for stream/filter. However, as a regular user you can only follow up to 5000 users (but you should get all of their tweets). If this limits you, you can apply for Shadow-role to up this limit to more users (20k afair). To answer your question: thousands of tweets per hour should be doable, as long as you don't need to follow more than 5000 users. -d -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Optimizing Twitter Calls
On Wed, Nov 10, 2010 at 11:08 AM, mage26 mag...@gmail.com wrote: I am currently integrating Twitter with a small social network. I am using PHP and the twitteroauth library. The idea is that a user can sync their account to their Twitter. So when they post on their wall, it will send a tweet and it will post their tweets onto their wall. Currently, I am gathering user tweets via a PHP script run by cron that runs like this: 1. go through every user that has connected to their twitter account 2. pull the last 20 tweets for each user. 3. See if any of these tweets are new. If so, deposit into database. You could easily replace this by using the streaming API. No rate limiting no cron job, just works. --d -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Random 403 Denied due to update limit errors.
You mention 150/hour so I am guessing that you are not authenticating with Twitter. This means that if there are any other applications sharing the same IP address (shared hosting) then they are also using the same rate limit as your app. If you authenticate you will get the 250/hour ratelimit and it will be all yours to consume. Dave Hi Dave, When I read Peter's post, I thought of the same thing. However, since he is successfully posting, wouldn't that require to be authenticated? --d -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Entities display_url and expanded_url
Is the expanded_url field only intended to be present for t.co- shortened links, or will it be extended to work with bit.ly and other services? Afaik it's only for t.co. To get bit.ly and other shortend url's resolved I use http://long-shore.com/ They have a convenient easy-to-use api -d -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Extract a one month social graph around a topic
Hi, Hi, I am new to the Twitter API world and appreciate your help. I am trying to do some sentiment analysis around few topics. The options that I think I have (although dont know if these options actually exist) 1. download last one month of tweets (related to various topic) and then run sentiment analysis on them or 2. run the Search API (w/ query = topic) several times in a day and collect the tweets over a period of month and then do the data analysis. the problem with this method is that twitter rate limits how many search query one can make in a day (white-listing seems to be taking ages) I am sure several people have done similar things in past. would really appreciate if you could please share your thoughts/experience. Ad 1. afaik you cannot just download the past month from twitter. Search results only go back a couple (~5) of days, depending of traffic. Ad 2. If you're not interested in past events that might work. However, I'd advise using the Streaming API with the track parameter. This should do exactly what you want and you don't have to query multiple times, and thus risk to be banned/blacklisted. cheers -d -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Where to submit a bug report?
Hi, I don't know any other way to get the statuses of approximately 60 users every minute when my app is behind a firewall (preventing OAuth authentication for each individual user). So I use my app's access token/secret to get my friends (and their most recent status). This sounds like it should be trivially be done with the streaming api. cheers -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Error posting _ to twitter using statuses/update
interesting, posting as a status message throught the api (python-twitter) worked for me. HTTP/1.1 401 Unauthorized maybe sth. else is broken? -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
Re: [twitter-dev] Client hourly limitations
On Thu, Sep 16, 2010 at 1:57 PM, Mike Southern gb1...@cox.net wrote: I have several twitter accounts. Each account has generated its own set of OAUTH credentials. I am just using to look at data of those who are following me based on a list of all my followers IDs. I understand that there is a 150 per hour limit on client requests. But after reaching 150 with my first twitter account credentials, I then get this message for all accounts, with different credentials. hash request /1/friendships/show.xml?source_id=110236585target_id=132286315 /request error Rate limit exceeded. Clients may not make more than 150 requests per hour. /error /hash So what is being checked in order to determine the number of times a client has made a request? I also need confirmation on source_id and target_id. I am assuming *I* am the source_id and that my follower is therefore the target_id ... Mike Hi Mike, I think you'd be best off getting your follower_ids and using the stream API to track them (up to 5000 users). Then you don't have the rate limit anymore. cheers -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
Re: [twitter-dev] Client hourly limitations
Check AnyEvent::Twitter::Stream for connecting to Streaming in Perl. It doesn't do oAuth yet, though. Not sure that oAuth is important here (unless your followers are protected users). At least I can get the followers of public users and create a stream based on these without using oAuth (unless you can do oAuth unknowingly, which would be sad since I cannot get oAuth to work willingly) cheers d -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
[twitter-dev] Fwd: REST API Server Headers wrong?
Hello, (sorry for the resend but i did not see this message appear on the list) It occurs from time to time that the rest api servers don't deliver responses, and I'm not complaining about that. However, why would the server answer with a 500 Http status code, a content type of application/json and the content being the html error page. E.g., GET /1/statuses/friends.json?cursor=1346292258256227567user_id=33307124 HTTP/1.1 Accept-Encoding: identity Host: api.twitter.com Connection: close User-Agent: Python-urllib/2.6 HTTP/1.1 500 Internal Server Error Date: Sat, 11 Sep 2010 01:56:02 GMT Server: hi Status: 500 Internal Server Error X-Transaction: 1284170160-87257-46051 X-RateLimit-Limit: 2 Last-Modified: Sat, 11 Sep 2010 01:56:00 GMT X-RateLimit-Remaining: 11250 Content-Type: application/json; charset=utf-8 Content-Length: 4659 Pragma: no-cache Connection: close !DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd; html xmlns=http://www.w3.org/1999/xhtml; lang=en xml:lang=en head meta http-equiv=Content-Type content=text/html; charset=utf-8 / meta http-equiv=Content-Language content=en-us / titleTwitter / Error/title link href=http://s.twimg.com/images/favicon.ico; rel=shortcut icon type=image/x-icon / thx -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
[twitter-dev] REST API Server Headers wrong?
Hello, It occurs from time to time that the rest api servers don't deliver responses, and I'm not complaining about that. However, why would the server answer with a 500 Http status code, a content type of application/json and the content being the html error page. E.g., GET /1/statuses/friends.json?cursor=1346292258256227567user_id=33307124 HTTP/1.1 Accept-Encoding: identity Host: api.twitter.com Connection: close User-Agent: Python-urllib/2.6 HTTP/1.1 500 Internal Server Error Date: Sat, 11 Sep 2010 01:56:02 GMT Server: hi Status: 500 Internal Server Error X-Transaction: 1284170160-87257-46051 X-RateLimit-Limit: 2 Last-Modified: Sat, 11 Sep 2010 01:56:00 GMT X-RateLimit-Remaining: 11250 Content-Type: application/json; charset=utf-8 Content-Length: 4659 Pragma: no-cache Connection: close !DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd; html xmlns=http://www.w3.org/1999/xhtml; lang=en xml:lang=en head meta http-equiv=Content-Type content=text/html; charset=utf-8 / meta http-equiv=Content-Language content=en-us / titleTwitter / Error/title link href=http://s.twimg.com/images/favicon.ico; rel=shortcut icon type=image/x-icon / thx -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
Re: [twitter-dev] Status format stream API vs. REST API
You can obtain entities from most API methods that return tweets by appending an ?include_entities=true parameter to the request. Eventually, entities should be part of the default response. You can read more about entities here: http://dev.twitter.com/pages/tweet_entities Methods that should support this query parameter generally indicate compatibility on the documentation page corresponding to the resource. Including entities on the REST API can sometimes increase total processing time, so if you're asking for a large amount of data, you might want to lower the total count you ask for at a time so that your request doesn't time out. Thanks, Taylor On Wed, Sep 8, 2010 at 3:17 PM, yaemog Dodigo yae...@gmail.com wrote: Hello, I'm wondering why the format of status messages that get delivered via the stream API differs from status elements that are retrieved via the REST API (e.g., public_timeline or user_timeline). More precisely, status elements from the stream API contain an 'entities' object with user_mentions, hashtags, and urls properties. This information is missing from status messages that are retrieved from the REST API. Are there plans to unify these formats? Hi, as of this morning I don't see entities anymore in the spritzer stream ( http://stream.twitter.com/1/statuses/sample.json). Is this expected behavior? Appending include_entities=true does not help either. thx -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
[twitter-dev] Status format stream API vs. REST API
Hello, I'm wondering why the format of status messages that get delivered via the stream API differs from status elements that are retrieved via the REST API (e.g., public_timeline or user_timeline). More precisely, status elements from the stream API contain an 'entities' object with user_mentions, hashtags, and urls properties. This information is missing from status messages that are retrieved from the REST API. Are there plans to unify these formats? thx -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
Re: [twitter-dev] How to access the twitter-streams
I guess I found it... http://stream.twitter.com/1/statuses/sample.json is this the spritzer stream? I got it from: http://dev.twitter.com/pages/streaming_api_methods#methods what's confusing is that the page talks about Firehose and Gardenhose, but does not mention Spritzer thx fred On Wed, Sep 1, 2010 at 6:06 PM, fred yae...@gmail.com wrote: Hello, first off, yes I read the FAQs and yes I read the docs. However, the stream-documentation does not provide a url to connect to and google results indicate that one should connect to http://betastream.twitter.com/spritzer.json which only returns: http://apiwiki.twitter.com/Streaming-API-Documentation been there done that. any pointers? thx fred -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
