[twitter-dev] Re: Receiving streaming API tweets without id_str
I can confirm that I am also seeing this via the Site Streams API. We are checking for messages of type 'tweet' by checking for existence of 'id_str'. Occasionally, this is missing. -Craig On Nov 6, 5:08 pm, Adam Green 140...@gmail.com wrote: My error logs started showing tweets without an id_str value a few days ago. I investigated today and found that these tweets are coming about 5-6 times an hour out of about 500 tweets per hour. I am using Phirehose to gather these tweets from the streaming API. I am collecting the tweets in JSON format. Neither my code or Phirehose has changed in the recent past, and as I said, about 99% of the tweets are fine. Here is an example of a bad tweet. It has a truncated field, which I haven't seen before. I assume this has something to do with the missing values. Does anyone have any idea what is happening here? [new_id_str] = 951769790156802 [place] = [truncated] = [user] = stdClass Object ( [follow_request_sent] = [time_zone] = Jakarta [url] =http://lada-hitam.livejournal.com/ [profile_background_color] = B2DFDA [screen_name] = MissLadaHitam [profile_background_image_url] =http://s.twimg.com/a/1288660386/images/themes/theme13/bg.gif [profile_text_color] = 33 [listed_count] = 0 [lang] = en [profile_background_tile] = [statuses_count] = 891 [following] = [favourites_count] = 159 [profile_link_color] = 93A644 [show_all_inline_media] = [profile_use_background_image] = 1 [description] = [contributors_enabled] = [profile_sidebar_fill_color] = ff [protected] = [location] = Indonesia [geo_enabled] = [name] = Lada Hitam [notifications] = [friends_count] = 8 [profile_sidebar_border_color] = ee [id] = 117790534 [verified] = [utc_offset] = 25200 [created_at] = Fri Feb 26 16:36:22 + 2010 [followers_count] = 18 [profile_image_url] =http://a1.twimg.com/profile_images/740195225/42360__1__normal.jpg ) [in_reply_to_status_id] = [favorited] = [source] = web [new_id] = 9.517697901568E+14 [contributors] = [in_reply_to_screen_name] = [coordinates] = [retweet_count] = [in_reply_to_user_id] = [entities] = stdClass Object ( [user_mentions] = Array ( ) [hashtags] = Array ( ) [urls] = Array ( ) ) [geo] = [retweeted] = [id] = 9.517697901568E+14 [text] = Today's cookings mostly failed :( It makes me less motivated to try another recipe tomorrow. But I have some mushrooms that need to be used. [created_at] = Sat Nov 06 16:44:54 + 2010 ) -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Rate limit question (again/followup) 20k user or ip?
Also, the 'reset-time' is increasing with every request. My reset time below was taken at 7:57 UTC: ?xml version=1.0 encoding=UTF-8? hash remaining-hits type=integer2/remaining-hits hourly-limit type=integer2/hourly-limit reset-time-in-seconds type=integer1249981041/reset-time-in- seconds reset-time type=datetime2009-08-11T08:57:21+00:00/reset-time /hash And this one at 7:58 UTC: ?xml version=1.0 encoding=UTF-8? hash remaining-hits type=integer2/remaining-hits reset-time type=datetime2009-08-11T08:58:30+00:00/reset-time hourly-limit type=integer2/hourly-limit reset-time-in-seconds type=integer1249981110/reset-time-in- seconds /hash On Aug 11, 8:55 am, CaMason stasisme...@googlemail.com wrote: FYI, my whitelisting counts have been a bit flaky since the DDoS. On most of my whilelisted IPs, I'm sometimes seeing the remaining count decreasing, and sometimes not. In-fact, my logs show it didn't decrease for a good chunk of yesterday. There have probably been some temporary adjustments at Twitter's end, so I wouldn't assume that the current whitelist results are the norm. -Craig On Aug 11, 8:23 am, Robert Fishel bobfis...@gmail.com wrote: While this may be true I think it's a fringe case and not what we're trying to get at here (although it could explain conflicting test results) To summarize what we're looking for clarification on: (example) My server has 1 whitelisted IP and 1000 users. It operates for 1 hour. Each user makes an equal number of requests. Is the limit 20 requests per user (= 20k per hour per ip) or Is the limit 20k per user (=20k per hour per user) The only reason I'm kind of harping on this is that for the new app I'm developing the latter would save me a lot of heartache and quite a bit of money. Cheers, Bob On Tue, Aug 11, 2009 at 1:54 AM, TFT Mediatftmedia1...@gmail.com wrote: I believe sometimes the IP address can be user-based, even for white- listed IPs. E.G., if the user himself has a whitelisted IP. On Aug 10, 7:57 pm, Dewald Pretorius dpr...@gmail.com wrote: Jim, I don't know exactly what you're looking at and how you get to that answer. My system is making thousands of GET calls per hour, and I can see how X-RateLimit-Remaining is decrementing regardless of which Twitter user credentials are used. So, on my side I am seeing solid evidence that the rate limit is per IP address only and not per user. Dewald On Aug 10, 11:26 pm, jim.renkel james.ren...@gmail.com wrote: Hmmm! We seem to have conflicting evidence here! I just (again) verified that twxlate.com is getting 20k requests per hour per user. How long ago was it that Alex and other API team members made the recommendation that you mentioned? Is it possible that twitter changed policy since then? Either way, I agree that we now need a very clear affirmation from twitter as to the policy. I sure hope I don't have to eat my words! :-) Jim On Aug 10, 9:08 pm, Dewald Pretorius dpr...@gmail.com wrote: On Aug 10, 11:02 pm, jim.renkel james.ren...@gmail.com wrote: My logic is now: Ifratelimiting is not peruser, then all users of anIPaddress will share one pool of20krequests per hour. If a site has a 1,000 users at one time, then eachuserwill get an average of 20 requests per hour. This is clearly not enough to do much useful. Jim, That is why Alex and other API team members have recommended in the past that you get and use additional white-listedIPaddresses, when 20,000 requests per hour perIPaddress is not sufficient to service youruserbase. At TweetLater I employ several white-listedIPaddresses to cover the needs of my users. Dewald
[twitter-dev] Re: OK Seriously People
it's a clear message to us in the third-party ecosystem that we'd better not make them our primary focus because we can't rely on them being here tomorrow if things get really bad. Surely, that's a wise move anyway, considering Twitter is a third- party supplier of free data. All of us app developers are at the whim of Twitter's operations and business decisions. Relying on third-parties for any business is risky - especially ones that have no contractual obligations to you. On Aug 9, 4:35 pm, Dossy Shiobara do...@panoptic.com wrote: On 8/9/09 10:00 AM, Joe Bowman wrote: First off, to people stating that Twitter Ops needs to work 30 hour shifts, and any ops person who hasn't, isn't a real ops person. A real ops person knows that after about 15 hours their mental capacity to solve problems begins to deteriorate and they have to rest in order to not cause further damage to their organization. Maybe in the little mom and pop shops where you run the show you're stuck with those 30 hour marathons, but in larger shops you have team members, and you schedule with them. People didn't state it. I did. I've worked in Ops in small 3-person companies as well as for 5,000+ employee companies with an IT team of 60+ people. In both situations, there have been exceptional times where yes, I've taken a 30 minute nap on the floor to recharge through a 30+ hour marathon emergency situation. I'm not suggesting that these people don't need to sleep (unless they have access to some Provigil and/or Adderall), but if Twitter isn't treating this outage as a #1 priority - and I mean literally, not figuratively - then it's a clear message to us in the third-party ecosystem that we'd better not make them our primary focus because we can't rely on them being here tomorrow if things get really bad. Here's an interesting game to play: how many Twitter people (Ops or otherwise) do you think are actively working on fixing the problem at this very moment? (Don't include people just communicating or otherwise not able to directly affect the situation.) a) 0 b) 1-5 c) 6-10 d) 11-15 e) 16-20 f) 20+ I'm guessing (c), 6-10. And, that's me being optimistic for a change. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
[twitter-dev] Re: How do I handle 302 redirects with curl?
To confirm, I am also seeing this behaviour. Some output I've received on numerous occasions this evening: -bash-3.2# curl --interface eth0 http://twitter.com/account/rate_limit_status.xml !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/ TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML -Craig On Aug 8, 11:25 pm, Chad Etzel c...@twitter.com wrote: Hmm, it shouldn't be spitting back HTML. How often are you seeing this? -Chad On Sat, Aug 8, 2009 at 1:02 PM, Naveen Ayyagariknig...@gmail.com wrote: Sometimes the rate_limit_status call is not returning a 302 to redirect, or the rate_limit_status xml, but HTML with a meta refresh in it (which curl doesnt understand to follow redirect/retry). Its not huge problem for us, but it can affect some throttling code people may or may not be implementing. (when you get this response, a subsequent retry request usually succeeds 95% of the time) Here is an example response I am talking about: curlhttp://twitter.com/account/rate_limit_status.xml-L -v * About to connect() to twitter.com port 80 (#0) * Trying 168.143.162.68... connected * Connected to twitter.com (168.143.162.68) port 80 (#0) GET /account/rate_limit_status.xml HTTP/1.1 User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 Host: twitter.com Accept: */* * HTTP 1.0, assume close after body HTTP/1.0 200 OK Connection: Close Pragma: no-cache cache-control: no-cache Refresh: 0.1 Content-Type: text/html; charset=iso-8859-1 !DOCTYPE html PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd !-- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN http://www.w3.org/TR/html4/strict.dtd; -- HTML HEAD META HTTP-EQUIV=Refresh CONTENT=0.1 META HTTP-EQUIV=Pragma CONTENT=no-cache META HTTP-EQUIV=Expires CONTENT=-1 TITLE/TITLE /HEAD BODYP/BODY /HTML On Aug 8, 2009, at 11:20 AM, Chad Etzel wrote: Hmm, Unfortunately this 302 business will completely goof OAuth calls. If you are able to programmatically see that you are getting these redirects, try calling the account/rate_limit_status call [1] (it could be any call, but this one is free and is a GET). You should still get a 302 (I'm pretty sure). Then if you jump through the redirect hoops with this call, it should clear you from more 302's for a while. I'm out today, but if someone could try this and report back if it works that would be helpful. [1]http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0ra... Thanks, -Chad On Sat, Aug 8, 2009 at 6:31 AM, timwhitlocktim.whitl...@publicreative.com wrote: I've seem the 302 Location headers having invalid URLs... i.e. two ? symbols. The original query string and then an additional ? for the token at the end. Following this redirect blindly has resulted in a Forbidden response. Also it is unclear whether the redirect location needs to be re- signed? (I am not doing so, but may explain the 403?) On Aug 8, 8:14 am, Rich rhyl...@gmail.com wrote: Excellent our client now supports the 302's :) On Aug 8, 7:37 am, Chad Etzel c...@twitter.com wrote: You may have to follow redirects more than once *wink wink nudge nudge* with curl you can add --location flag. There's a good bit of info in the man page as well. If using curl with PHP, you can set: curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); HTH, -Chad On Sat, Aug 8, 2009 at 1:31 AM, TjLluo...@gmail.com wrote: All of my scripts check for Status 200 before proceeding. Now we are (sometimes) getting a 302, but when I try curl --netrc -s -D -http://twitter.com/account/rate_limit_status.xml Gave me a 302 with a Location of: http://twitter.com/account/rate_limit_status.xml?c73f7db0 but when I tried curl --netrc -s -D - 'http://twitter.com/account/rate_limit_status.xml?c73f7db0' it seemed to want to redirect me to http://twitter.com/account/rate_limit_status.xml If accepting 30x is a requirement now, I'd like some advice on how to do so. TjL
[twitter-dev] Re: Regex for @replies
It looks like they're simply applying this regex as a test: (?![\w])@username(?![\w]) Thus, if a character on either side is not (a-z, A-Z, 0-9, _) then it is a mention. any 'word' character (a-z, A-Z, 0-9, _) on either side of '@screenname' causes the mention to fail. (I hope I got the regex explanation correct!). -Craig On May 12, 12:33 pm, hjb ha...@heatonmoor.com wrote: @Doug, Is this behavour likely to remain? ( I noticed that @replies and - @replies are successful ) That is to say, I'm sure @replies will work at some point via sms, but can we rely on the fact that _...@replies do not? Is this related to there being any chance of it being an email address? Thanks, Harry On May 11, 6:26 pm, Doug Williams d...@twitter.com wrote: In my test posts @dougw and @DOUGW worked as mentions. t...@dougw and _...@dougw were not included as mentions. Thanks, Doug -- Doug Williams Twitter Platform Supporthttp://twitter.com/dougw On Mon, May 11, 2009 at 10:16 AM, CaMason stasisme...@googlemail.comwrote: Thanks Doug, that's a great help. How about preceding? i.e. should t...@dougw, _...@dougw or @dougw create mentions? The main concern here obviously is email addresses. And finally, are screen names case sensitive? :) Cheers On May 11, 6:07 pm, Doug Williams d...@twitter.com wrote: The classic definition of an @reply is any tweet that starts with @user. If you perfrom a to:user (e.g. to:dougw) query at search.twitter.com you will only get @replies. @replies were converted to mentions after we realized people didn't just @reply. Mentions are any tweet that contain @user within the text of the tweet. So @replies are a subset of mentions. Any non-alphanumeric (where alphanumeric is a-z, 0-9, or _) can terminate the username. For instance: hi @dougw, you look dapper today is a mention. Thanks, Doug -- Doug Williams Twitter Platform Supporthttp://twitter.com/dougw On Mon, May 11, 2009 at 2:36 AM, stasisme...@googlemail.com stasisme...@googlemail.com wrote: Hi guys, For an application I'm working on, we have a single table for 'tweets' and another for DMs. We're linking TwitterUsers to Tweets with a many:many, and a simple flag to specify if the tweet is a reply/ mention. We first pull in messages from the user_timeline feed, then the mentions feed. As such, we'd like to check if any of the messages in user_timeline feed is actually a reply. Could anybody clarify the exact rules that are used to determine whether a string is a reply/mention? i.e. preceded by start-of-string or non-word character... followed by space, comma, period or end of message... case insensitive... [not even sure if these are correct! :) ] Currently I'm using: /(?![^\W_])@%s(?![^\W_])/i with %s replaced by the user's screen name. Perhaps one of the devs could share the exact rules (or even the regex), or propose a nicer mechanism for detecting replies. (I did propose checking for replies before tweets, but these update threads are run asynchronously). Cheers
[twitter-dev] Re: Regex for @replies
Thanks Doug, that's a great help. How about preceding? i.e. should t...@dougw, _...@dougw or @dougw create mentions? The main concern here obviously is email addresses. And finally, are screen names case sensitive? :) Cheers On May 11, 6:07 pm, Doug Williams d...@twitter.com wrote: The classic definition of an @reply is any tweet that starts with @user. If you perfrom a to:user (e.g. to:dougw) query at search.twitter.com you will only get @replies. @replies were converted to mentions after we realized people didn't just @reply. Mentions are any tweet that contain @user within the text of the tweet. So @replies are a subset of mentions. Any non-alphanumeric (where alphanumeric is a-z, 0-9, or _) can terminate the username. For instance: hi @dougw, you look dapper today is a mention. Thanks, Doug -- Doug Williams Twitter Platform Supporthttp://twitter.com/dougw On Mon, May 11, 2009 at 2:36 AM, stasisme...@googlemail.com stasisme...@googlemail.com wrote: Hi guys, For an application I'm working on, we have a single table for 'tweets' and another for DMs. We're linking TwitterUsers to Tweets with a many:many, and a simple flag to specify if the tweet is a reply/ mention. We first pull in messages from the user_timeline feed, then the mentions feed. As such, we'd like to check if any of the messages in user_timeline feed is actually a reply. Could anybody clarify the exact rules that are used to determine whether a string is a reply/mention? i.e. preceded by start-of-string or non-word character... followed by space, comma, period or end of message... case insensitive... [not even sure if these are correct! :) ] Currently I'm using: /(?![^\W_])@%s(?![^\W_])/i with %s replaced by the user's screen name. Perhaps one of the devs could share the exact rules (or even the regex), or propose a nicer mechanism for detecting replies. (I did propose checking for replies before tweets, but these update threads are run asynchronously). Cheers
[twitter-dev] Re: Freelance Twitter API Dev directory?
Why not... twitter id: twitter.com/CraigMason company: Stasis Media (http://www.stasismedia.com) email: i...@stasismedia.com -Craig On Feb 23, 10:34 pm, Lakshman Prasad scorpion...@gmail.com wrote: Ok, If you are starting it anyway, Please include me in that. Happy to hack on twitter API, anyday! twitter id: twitter.com/scorpion032 Company: uswaretech.com/ Email: laksh...@uswaretech.com Thanks Alex, in advance! On Tue, Feb 24, 2009 at 12:03 AM, Alex Payne a...@twitter.com wrote: There isn't one that I'm aware of, but if people would like to post their contact info in this thread (Twitter username, URL, email, whatever) I'm happy to collect them on the API Wiki. On Sat, Feb 21, 2009 at 18:00, Chad Etzel jazzyc...@gmail.com wrote: Hi All, I have been getting a few requests here and there for twitter API development work. I cannot take on any such projects at the moment, but I always feel bad for leaving them in the lurch. Is there a list or directory anywhere of Twitter API developers that work freelance that I can send to them when this happens? I'm happy to forward on such requests. -Chad -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x -- Regards, Lakshman becomingguru.com lakshmanprasad.com
