[twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
It actually makes sense to use tweet ID to sort tweets, because timestamp is not a valid source of information for accurate sorting. It is a very common case to have multiple tweets posted at the exact same second, and it is not possible to reproduce the correct ordering of tweets on the client side. This can be improved by having better precision for timestamp (maybe milliseconds), but it is still possible to get tweets posted at the exact same milliseconds (although it is very rare). If Twitter really needs to change the tweet ID scheme, I think better solution for sorting is required to be provided through API. On Mar 27, 7:41 am, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Developers, It's no secret that Twitter is growing exponentially. The tweets keep coming with ever increasing velocity, thanks in large part to your great applications. Twitter has adapted to the increasing number of tweets in ways that have affected you in the past: We moved from 32 bit unsigned integers to 64-bit unsigned integers for status IDs some time ago. You all weathered that storm with ease. The tweetapoclypse was averted, and the tweets kept flowing. Now we're reaching the scalability limit of our current tweet ID generation scheme. Unlike the previous tweet ID migrations, the solution to the current issue is significantly different. However, in most cases the new approach we will take will not result in any noticeable differences to you the developer or your users. We are planning to replace our current sequential tweet ID generation routine with a simple, more scalable solution. IDs will still be 64-bit unsigned integers. However, this new solution is no longer guaranteed to generate sequential IDs. Instead IDs will be derived based on time: the most significant bits being sourced from a timestamp and the least significant bits will be effectively random. Please don't depend on the exact format of the ID. As our infrastructure needs evolve, we might need to tweak the generation algorithm again. If you've been trying to divine meaning from status IDs aside from their role as a primary key, you won't be able to anymore. Likewise for usage of IDs in mathematical operations -- for instance, subtracting two status IDs to determine the number of tweets in between will no longer be possible. For the majority of applications we think this scheme switch will be a non-event. Before implementing these changes, we'd like to know if your applications currently depend on the sequential nature of IDs. Do you depend on the density of the tweet sequence being constant? Are you trying to analyze the IDs as anything other than opaque, ordered identifiers? Aside for guaranteed sequential tweet ID ordering, what APIs can we provide you to accomplish your goals? Taylor Singletary Developer Advocate, Twitterhttp://twitter.com/episod -- To unsubscribe, reply using remove me as the subject.
[twitter-dev] When does DM rate limit reset?
Hi I know there is 250 rate limits to post DM in a day. 1) when does the rate limit reset? Does it reset to zero at 24:00 in PST every day? 2) If the user keeps on trying to send DM after exceeding the rate limit, is there a possibility that the account will be suspended or treated as spam? My client hits to the rate limit on twitter.com, but they didn't know about the rate limit. So they tried to send DM again and again. Thanks Aki -- To unsubscribe, reply using remove me as the subject.
[twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
I'm developing desktop Twitter client. I think accurate sorting is needed, because the order of tweets may look different on every application without accurate sorting. It's not that it would totally kill my Twitter client, but I take accurate presentation of tweets seriously, and I think it would be better to have consistent tweet ordering across all applications. If this scheme change is really needed (e.g. required to processing new tweets simultaneously across multiple servers without synchronising tweet ID), I would suggest adding time in milliseconds to tweet information, which would have much better accuracy. On Apr 2, 3:39 am, Mark McBride mmcbr...@twitter.com wrote: Just out of curiosity, what applications are you building that require sub-second sorting resolution for tweets? ---Mark http://twitter.com/mccv On Wed, Mar 31, 2010 at 11:01 PM, Aki yoru.fuku...@gmail.com wrote: It actually makes sense to use tweet ID to sort tweets, because timestamp is not a valid source of information for accurate sorting. It is a very common case to have multiple tweets posted at the exact same second, and it is not possible to reproduce the correct ordering of tweets on the client side. This can be improved by having better precision for timestamp (maybe milliseconds), but it is still possible to get tweets posted at the exact same milliseconds (although it is very rare). If Twitter really needs to change the tweet ID scheme, I think better solution for sorting is required to be provided through API. On Mar 27, 7:41 am, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Developers, It's no secret that Twitter is growing exponentially. The tweets keep coming with ever increasing velocity, thanks in large part to your great applications. Twitter has adapted to the increasing number of tweets in ways that have affected you in the past: We moved from 32 bit unsigned integers to 64-bit unsigned integers for status IDs some time ago. You all weathered that storm with ease. The tweetapoclypse was averted, and the tweets kept flowing. Now we're reaching the scalability limit of our current tweet ID generation scheme. Unlike the previous tweet ID migrations, the solution to the current issue is significantly different. However, in most cases the new approach we will take will not result in any noticeable differences to you the developer or your users. We are planning to replace our current sequential tweet ID generation routine with a simple, more scalable solution. IDs will still be 64-bit unsigned integers. However, this new solution is no longer guaranteed to generate sequential IDs. Instead IDs will be derived based on time: the most significant bits being sourced from a timestamp and the least significant bits will be effectively random. Please don't depend on the exact format of the ID. As our infrastructure needs evolve, we might need to tweak the generation algorithm again. If you've been trying to divine meaning from status IDs aside from their role as a primary key, you won't be able to anymore. Likewise for usage of IDs in mathematical operations -- for instance, subtracting two status IDs to determine the number of tweets in between will no longer be possible. For the majority of applications we think this scheme switch will be a non-event. Before implementing these changes, we'd like to know if your applications currently depend on the sequential nature of IDs. Do you depend on the density of the tweet sequence being constant? Are you trying to analyze the IDs as anything other than opaque, ordered identifiers? Aside for guaranteed sequential tweet ID ordering, what APIs can we provide you to accomplish your goals? Taylor Singletary Developer Advocate, Twitterhttp://twitter.com/episod -- To unsubscribe, reply using remove me as the subject.
[twitter-dev] Re: API Limit of 150 is Obsolete
I would also like to request the API limit to be raised as soon as possible. I am not able to roll out an update for my desktop Twitter client, simply because the API limit is too low for the features I have implemented. The new version of my client utilizes Lists related APIs, which consumes a lot of API calls. Moreover, multiple API calls are currently required to get a chain of replies (conversation), which consumes significant number of API calls. The previously discussed features were implemented, because there was an official announcement to change the API limit to 1500. I know it is expected to happen in few weeks, but it would be good to know the estimated date for this, so the developers can make appropriate decisions on the application deployment timeline. -- Aki On Jan 21, 9:48 am, Eric Woodward e...@nambu.com wrote: I will come straight to the point: we need to an increase to the API limit to properly implement Twitter within a desktop client application given the addition of: 1) three retweets timelines; 2) checking the account's saved searches; and 3) up to 10-20 Twitter Lists timelines. Twitter Lists alone are causing real problems if a user follows more than 5 or so. We cant poll Twitter List subscriptions with one API call that combines them altogether, which we could then split apart client-side with some attached meta-data. That alone would have been a big help, and without it we are left polling each List as if it was a separate timeline, since that is what they are. Implementing proper Lists management is a non-starter within this limit, so is regular confirmation of a relationship between two users when asked for by the user (on Lists or search results). There is simply a lot of stuff I cannot do properly that is standard on twitter.com, all because I am subject to the API limit while twitter.com is not. Users simply do not understand this distinction in possibilities. I would like to formally ask on behalf of all client developers that the API limit increase to 250, from 150, for all applications whether they use OAuth or HTTP Basic Authentication. We are simply not able to implement Twitter properly within a limit of 150, but dont need a lot more, only another 100-200 API calls or so. If Twitter can even technically contemplate a 10x API limit increase to 1,500 for OAuth applications, surely an increase to 250 based on the addition of core features like official retweets and Lists is a reasonable request. A limit of 150 is simply obsolete, and has been for a long time. I do not want to wait for the UX repairs around OAuth for desktop applications, and I dont like being forced into OAuth sooner than we are ready just because we need the extra API hits just to do basic features properly. And besides, that was announced as two weeks away three weeks ago. I dont want to wait any longer. I want to properly implement the basics, like Lists polling, now. This is a considered email because I care about the quality of our Twitter implementation and I care about the Twitter ecosystem. I would appreciate a considered reply. --ejw Eric Woodward Email: e...@nambu.com
[twitter-dev] Re: Social Graph API: Legacy data format will be eliminated 1/11/2010
I agree with PJB. The previous announcements only said that the
pagination will be deprecated.
1.
http://groups.google.com/group/twitter-api-announce/browse_thread/thread/41369cb133175d0f#
2.
http://groups.google.com/group/twitter-api-announce/browse_thread/thread/52d4e68040d4ca45#
However, both of the announcements did not say that the API call
without page parameter to get
all IDs will be removed or replaced with cursor pagination.
The deprecation of this method is not being documented as PJB said.
On Dec 24, 5:00 pm, PJB pjbmancun...@gmail.com wrote:
Why hasn't this been announced before? Why does the API suggest
something totally different? At the very least, can you please hold
off on deprecation of this until 2/11/2010? This is a new API change.
On Dec 23, 7:45 pm, Raffi Krikorian ra...@twitter.com wrote:
yes - if you do not pass in cursors, then the API will behave as though you
requested the first cursor.
Willhelm:
Your announcement is apparently expanding the changeover from page to
cursor in new, unannounced ways??
The API documentation page says: If the cursor parameter is not
provided, all IDs are attempted to be returned, but large sets of IDs
will likely fail with timeout errors.
Yesterday you wrote: Starting soon, if you fail to pass a cursor, the
data returned will be that of the first cursor (-1) and the
next_cursor and previous_cursor elements will be included.
I can understand the need to swap from page to cursor, but was pleased
that a single call was still available to return (or attempt to
return) all friend/follower ids. Now you are saying that, in addition
to the changeover from page to cursor, you are also getting rid of
this?
Can you please confirm/deny?
On Dec 22, 4:13 pm, Wilhelm Bierbaum wilh...@twitter.com wrote:
We noticed that some clients are still calling social graph methods
without cursor parameters. We wanted to take time to make sure that
people were calling the updated methods which return data with cursors
instead of the old formats that do not.
As previously announced in September (http://bit.ly/46x1iL) and
November (http://bit.ly/3UQ0LU), the legacy data formats returned
as a result of calling social graph endpoints without a cursor
parameter are deprecated and will be removed.
These formats have been removed from the API wiki since September.
You should always pass a cursor parameter. Starting soon, if you fail
to pass a cursor, the data returned will be that of the first cursor
(-1) and the next_cursor and previous_cursor elements will be included.
If you aren't seeing next_cursor and previous_cursor in your results,
you are getting data back in the old format. You will need to adjust
your parser to handle the new format.
We're going to start assuming you want data in the new format
(users_list / users / user or id_list / ids / id) instead of the old
format (users / user or ids / id) regardless of your passing a cursor
parameter as of 1/11/2010.
* The old formats will no longer be returned after 1/11/2010.
* Start using the new formats now by passing the 'cursor' parameter.
To recap, the old endpoints at
/statuses/friends.xml
/statuses/followers.xml
returned
users type=array
user
!-- ... omitted ... --
/user
/users
or JSON like [{/*user record*/ /*, .../]
whereas
/statuses/friends.xml?cursor=n
/statuses/followers.xml?cursor=n
return data that looks like
users_list
users type=array
user
!-- ... omitted ... --
/user
/users
next_cursor7128872798413429387/next_cursor
previous_cursor0/previous_cursor
/users_list
or, the JSON equivalent:
{users:[{/*user record*/} /*, ...*/], next_cursor:0,
previous_cursor:0}
and the old endpoints at
/friends/ids.xml
/followers/ids.xml
returned data that looks like
ids
id1/id
id2/id
id3/id
/ids
whereas
/friends/ids.xml?cursor=n
/followers/ids.xml?cursor=n
return data that looks like
id_list
ids
id1/id
id2/id
id3/id
/ids
next_cursor1288724293877798413/next_cursor
previous_cursor-1300794057949944903/previous_cursor
/id_list
or, the JSON equivalent:
{ids:[1, 2, 3], next_cursor:0, previous_cursor:0}
If you have any questions or comments, please feel free to post them
to twitter-development-talk.
Thanks!
--
Wilhelm Bierbaum
Twitter Platform Team
--
Raffi Krikorian
Twitter Platform Teamhttp://twitter.com/raffi
[twitter-dev] Re: Twitter Search Issues for Japanese
Thank you for responding so quickly! It's nice to know that the issues have been reported to the appropriate people. -- Aki On Oct 27, 12:35 am, John Kalucki jkalu...@gmail.com wrote: I've forwarded your message to the Japan product manager and to the engineer who first brought Japanese tokenizing to Search. -John Kaluckihttp://twitter.com/jkalucki Services, Twitter.com On Oct 26, 2:27 am, Aki yoru.fuku...@gmail.com wrote: Hi, I'm developing desktop Twitter application for Mac, targeting Japanese Twitterers. I've recently implemented a feature to interface the official Twitter Search API. It came to my attention that there are several severe issues with the current implementation of Twitter Search. Firstly, many tweets posted by Japanese Twitterers are not being indexed at all. Yes, I have read the notice regarding missing tweets from the search results. (http://help.twitter.com/forums/10713/entries/42646) From looking at several people I follow, about 50% of people are not being indexed at all. Actually, tweets were never indexed at all. Many of the other Japanese developers have also noticed these issues. I have tried using the locale option for the search API, but it didn't help much. Secondly, the operator to execute an exact phrase search (documented athttp://search.twitter.com/operators) does not work for Japanese query. I'm not sure if it's supposed to work with the API though. This issue is annoying, because search results returned from a query containing Japanese characters seems to return very fuzzy results. I get very irrelevant results, and even find it difficult to figure out why it even matched at all. I consider Twitter Search to be the major feature of Twitter, and I think correcting these issues will make Twitter much better web service for Japanese people. Thank you. -- Aki
[twitter-dev] Twitter Search Issues for Japanese
Hi, I'm developing desktop Twitter application for Mac, targeting Japanese Twitterers. I've recently implemented a feature to interface the official Twitter Search API. It came to my attention that there are several severe issues with the current implementation of Twitter Search. Firstly, many tweets posted by Japanese Twitterers are not being indexed at all. Yes, I have read the notice regarding missing tweets from the search results. (http://help.twitter.com/forums/10713/entries/42646) From looking at several people I follow, about 50% of people are not being indexed at all. Actually, tweets were never indexed at all. Many of the other Japanese developers have also noticed these issues. I have tried using the locale option for the search API, but it didn't help much. Secondly, the operator to execute an exact phrase search (documented at http://search.twitter.com/operators) does not work for Japanese query. I'm not sure if it's supposed to work with the API though. This issue is annoying, because search results returned from a query containing Japanese characters seems to return very fuzzy results. I get very irrelevant results, and even find it difficult to figure out why it even matched at all. I consider Twitter Search to be the major feature of Twitter, and I think correcting these issues will make Twitter much better web service for Japanese people. Thank you. -- Aki
