Re: [twitter-dev] Re: Snowflake: An update and some very important information
Signed bigint is what Snowflake itself uses. Tom On 12/3/10 7:26 PM, TCI wrote: Help. I had been using BINARY(16) to store tweet id's in MYSQL. I can't find a good resource to tell me the new data type I should use: Reading the mysql documentation, I believe it should be unsigned bigint - but would binary(64) be a better option? I also read this comment against about BIGINT, with minimal comments. http://blog.salientdigital.com/2010/11/13/twitter-api-snowflake-and-mysql/ Anybody storing tweet id's in MYSQL care to share what datatype they're using? TCI -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Snowflake: An update and some very important information
1) Please don't, I don't want to have to convert everything back to
integers within my code. I consider the string representation a hack
around some issues with certain programming languages, and not an
optimal solution. Wouldn't want this to become the default option.
2) No
Tom
On 11/11/10 6:34 AM, SM wrote:
Hello. Couple questions:
1.) Are you planning on eventually eliminating the integer
representation and only using strings for id's?
2.) If an application doesn't use Javascript to parse JSON (for
example, YAJL-OBJC and NSNumbers in Obj-C), is it necessary to make
any changes at all?
Thanks.
On Oct 19, 3:52 pm, Matt Harris thematthar...@twitter.com wrote:
Hey everyone,
Thank you to all of you for your questions, patience and contributions to
this thread. Hearing your views and knowing how you use the API helps us
provide more information where there wasn't enough, and clarify details
where there was ambiguity.
I've collated the questions i've received from you directly, over Twitter to
@twitterapi and through this list. I hope the comments below provide enough
information to answer those questions and explain the reasoning being our
decisions.
Thanks for your support and patience,
@themattharris
1) Will search.twitter.com also include id_str and
in_reply_to_status_id_str?
Yes, Search will include the String representations of those IDs.
2) Which fields are affected by this change?
All IDs which are transmitted as Integers will have a String representation
in the API response. Only Tweet IDs (which includes mentions and retweets)
will be moving to new Snowflake IDs. Messages (DMs), Saved Searches and
Users may change to a Snowflake ID scheme in the future but this isn’t
planned for this year.
We are adding String representations of the Integer IDs now so you can
update all of your code to use the String representations throughout. to
allow developers to make the change now for all the ID fields and be
prepared should any other IDs break the 53bit boundary.
3) Which fields will have String representations?
The fields which will have String representations are:
id (DM, Saved Search, User, List )
in_reply_to_status_id
in_reply_to_user_id
new_id (Streaming only. Will be removed when Snowflake is enabled)
current_user_retweet_id (When include_my_retweet=1 is passed)
4) Can you provide a complete Tweet example with Snowflake ID to test?
[{coordinates:null,truncated:false,created_at:Thu Oct 14 22:20:15
+
2010,favorited:false,entities:{urls:[],hashtags:[],user_mentions
:[{name:Matt
Harris,id:777925,id_str:777925,indices:[0,14],screen_name:thema
ttharris}]},text:@themattharris
hey how are
things?,annotations:null,contributors:[{id:819797,id_str:819797,
screen_name:episod}],id:10765432100123456789,id_str:10765432100123
456789,retweet_count:0,geo:null,retweeted:false,in_reply_to_user_id
:777925,in_reply_to_user_id_str:777925,in_reply_to_screen_name:them
attharris,user:{id:6253282,id_str:6253282},source:web,place:
null,in_reply_to_status_id:10586268426842688951,in_reply_to_status_id_st
r:10586268426842688951}]
5) What is happening with new_id in the Streaming API?
new_id and new_id_str will be switched off when or soon after Snowflake is
enabled on November 4th.
6) Why not restrict IDs to 53bits?
A Snowflake ID is composed:
* 41bits for millisecond precision time (69 years)
* 10bits for a configured machine identity (1024 machines)
* 12bits for a sequence number (4096 per machine)
The factor influencing the length of the ID is the time. For a 53bit ID this
would mean only 31bits are available for the time. 31bits is only enough for
24 days (2147483648/(1000*60*60*24)) of time.
Reducing the resolution of the timestamp would prevent a K-sorted resolution
of 1 second or less.
Reducing the configured machine identity or sequence number by 1bit would
mean we couldn’t scale Twitter, or operate our infrastructure in an
uncoordinated high-available way.
7) When will the 53bit Integer overflow happen?
24 days after Snowflake starts counting.
8) Is it safe to parse and store IDs as signed 64bit Integers?
Yes.
9) Why offer both the String and Integer versions of the ID?
The String representation is needed to ensure languages which cannot convert
the 53bit Integer can still use the ID in other API requests.
The Integer value is being retained for languages which can handle
numbers53bit and to prevent applications which have not converted from being
cut-off from Twitter.
10) When ID is null what will the _str representation be?
The _str representation will also be null.
11) Did you really mean ‘unsigned’ 64bit Integer?
Strictly speaking the Snowflake is a signed 64bit long under the hood. That
being said, we will never use the negative bit and won’t require the full
64bits for positive numbers for about 69 years:
Re: [twitter-dev] Re: Snowflake: An update and some very important information
It's there. But for some reason now in all types of json requests -
On Oct 25, 2010, at 6:28 AM, Johannes la Poutre wrote:
@Themattharris: was there any change to the implementation timeline?
Quote: by 22nd October 2010 (Friday): String versions of ID numbers
will start appearing in the API responses
I'm still not seeing id_str, to_user_id_str and from_user_id_str etc.
in the current search API output, example:
http://search.twitter.com/search.json?geocode=52.155018%2C4.487658%2C1km
is there an updated timeline or did I miss something?
Best,
-- Johannes / @jlapoutre / @tweepsaround
On Oct 19, 2:19 am, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status
ID
generator - 'Snowflake' but didn't. The purpose of this email is to
explain
the reason why this didn't happen, what we are doing about it, and
what the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet
IDs. These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is
composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming
languages such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to:
(90071992547409921).toString() in
your browsers console or by running the following JSON snippet
through your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully
and will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to
include a
string version of any ID when responding in the JSON format. What
this means
is Status, User, Direct Message and Saved Search IDs in the Twitter
API will
now be returned as an integer and a string in JSON responses. This
will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str.
The
following JSON representation of a status object shows the two
versions of
the ID fields for each data point.
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524
in_reply_to_status_id_str: 12738040524
}
]
What should you do - RIGHT NOW
--
The first thing you should do is attempt to decode the JSON snippet
above
using your production code parser. Observe the output to confirm
the ID has
not lost accuracy.
What you do next depends on what happens:
* If your code converts the ID successfully without losing accuracy
you are
OK but should consider converting to the _str versions of IDs as
soon as
possible.
* If your code has lost accuracy, convert your code to using the _str
version immediately. If you do not do this your code will be unable
to
interact with the Twitter API reliably.
* In some language parsers, the JSON may throw an exception when
reading the
ID value. If this happens in your parser you will need to ‘pre-
parse’ the
data, removing or replacing ID parameters with their _str versions.
Summary
-
1) If you develop in Javascript, know that you will have to update
your code
to read the string version instead of the integer version.
2) If you use a JSON decoder, validate that the example JSON,
above, decodes
without throwing exceptions. If exceptions are thrown, you will
need to
pre-parse the data. Please let us know the name, version, and
language of
the parser which throws the exception so we can investigate.
Timeline
---
by 22nd October 2010 (Friday): String versions of ID numbers will
start
appearing in the API responses
4th November 2010 (Thursday) : Snowflake will be turned on but at
~41bit
length
26th November 2010 (Friday) : Status IDs will break
Re: [twitter-dev] Re: Snowflake: An update and some very important information
Unless I did the math wrong, a 64 bit quantity is expressable in (64 * log(2)) / log(62) = 10.7487219 eleven characters drawn from A-Za-z0-9 and they can still be sortable! -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Snowflake: An update and some very important information
6) Why not restrict IDs to 53bits? A Snowflake ID is composed: * 41bits for millisecond precision time (69 years) * 10bits for a configured machine identity (1024 machines) * 12bits for a sequence number (4096 per machine) The factor influencing the length of the ID is the time. For a 53bit ID this would mean only 31bits are available for the time. 31bits is only enough for 24 days (2147483648/(1000*60*60*24)) of time. Reducing the resolution of the timestamp would prevent a K-sorted resolution of 1 second or less. Reducing the configured machine identity or sequence number by 1bit would mean we couldn’t scale Twitter, or operate our infrastructure in an uncoordinated high-available way. Interesting ... so you have the theoretical capacity to scale to 2**22 (about 4 million) tweets per millisecond? Even 4 million tweets a second seems unrealistic, as does a single machine only being able to generate 4096 IDs. I think if you're really expecting this kind of volume, the FPGA vendors probably can help you out. We are talking clocks and counters, here, right, not Javascript interpreters or robust linear regressions? ;-) Ah, well, I'll check back on you guys in 69 years to see how you're holding up. ;-) -- M. Edward (Ed) Borasky http://borasky-research.net http://twitter.com/znmeb A mathematician is a device for turning coffee into theorems. - Paul Erdos -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Snowflake: An update and some very important information
Yes, you can. Tom On Oct 20, 2010, at 1:37 PM, Navin Kabra navin.ka...@gmail.com wrote: On Oct 19, 5:19 am, Matt Harris thematthar...@twitter.com wrote: So what is Snowflake? -- Snowflake is a service we will be using to generate unique Tweet IDs. These Tweet IDs are unique 64bit unsigned integers, which, instead of being sequential like the current IDs, are based on time. The full ID is composed of a timestamp, a worker number, and a sequence number. Will the new IDs be monotonically increasing? If my programming language can handle 64bit integers, can I compare two IDs and expect that the lower ID represents an older tweet (approximately)? -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Snowflake: An update and some very important information
Isn't the point of having versioned API's so changes can be rolled out w/o breaking a much of applications at once? Why not increment to version 2 and replace all ID's as strings in the JSON format? Keep version 1 around for a few months allowing everyone to upgrade and then kill it off. This can also give twitter a chance to make any other breaking changes. If Twitter is never going to take advantage of the versioning they added what is the point of having it? I think just creating new fields to avoid versioning issues is unclean and messy. -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Snowflake: An update and some very important information
Because a lot of Twitter applications (including my own) would go crazy
immediately.
Tom
On 10/19/10 6:09 AM, Detroitpro wrote:
Out of curiosity; what advantage is there of including both the int
and the string? Why not only offer the string and put the parsing on
the client side?
Thanks,
@detroitpro
On Oct 18, 8:34 pm, themattharris thematthar...@twitter.com wrote:
Thanks to @gotwalt for spotting the missing commas.
Fixed JSON sample ...
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282,
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524,
in_reply_to_status_id_str: 12738040524
}
]
Best,
@themattharris
On Oct 18, 5:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to explain
the reason why this didn't happen, what we are doing about it, and what the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs. These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString() in
your browsers console or by running the following JSON snippet through your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to include a
string version of any ID when responding in the JSON format. What this means
is Status, User, Direct Message and Saved Search IDs in the Twitter API will
now be returned as an integer and a string in JSON responses. This will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str. The
following JSON representation of a status object shows the two versions of
the ID fields for each data point.
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524
in_reply_to_status_id_str: 12738040524
}
]
What should you do - RIGHT NOW
--
The first thing you should do is attempt to decode the JSON snippet above
using your production code parser. Observe the output to confirm the ID has
not lost accuracy.
What you do next depends on what happens:
* If your code converts the ID successfully without losing accuracy you are
OK but should consider converting to the _str versions of IDs as soon as
possible.
* If your code has lost accuracy, convert your code to using the _str
version immediately. If you do not do this your code will be
Re: [twitter-dev] Re: Snowflake: An update and some very important information
Probably null as well.
Tom
On 10/19/10 11:49 AM, Reivax wrote:
Hi
In the case where an id is null (as in in_reply_to_status_id:null )
what will the value of in_reply_to_status_id_str be ?
Thanks
Xavier
On 19 oct, 02:34, themattharris thematthar...@twitter.com wrote:
Thanks to @gotwalt for spotting the missing commas.
Fixed JSON sample ...
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282,
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524,
in_reply_to_status_id_str: 12738040524
}
]
Best,
@themattharris
On Oct 18, 5:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to explain
the reason why this didn't happen, what we are doing about it, and what the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs. These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString() in
your browsers console or by running the following JSON snippet through your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to include a
string version of any ID when responding in the JSON format. What this means
is Status, User, Direct Message and Saved Search IDs in the Twitter API will
now be returned as an integer and a string in JSON responses. This will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str. The
following JSON representation of a status object shows the two versions of
the ID fields for each data point.
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524
in_reply_to_status_id_str: 12738040524
}
]
What should you do - RIGHT NOW
--
The first thing you should do is attempt to decode the JSON snippet above
using your production code parser. Observe the output to confirm the ID has
not lost accuracy.
What you do next depends on what happens:
* If your code converts the ID successfully without losing accuracy you are
OK but should consider converting to the _str versions of IDs as soon as
possible.
* If your code has lost accuracy, convert your code to using the _str
version immediately. If you do not do this your code will be unable to
interact with the Twitter API reliably.
* In some language parsers, the JSON may throw an
Re: [twitter-dev] Re: Snowflake: An update and some very important information
I'm also patiently awaiting a response from twitter about this. Are the ids
sane for 64-bit *signed* long?
Dan
On Mon, Oct 18, 2010 at 9:08 PM, jon jonhoff...@gmail.com wrote:
Hi,
You wrote that the IDs are unsigned 64 bit ints, but the IdWorker is
pumping out java Longs which are signed. I'm assuming that was a
typo, but please clarify.
http://github.com/twitter/snowflake/blob/master/src/main/scala/com/twitter/service/snowflake/IdWorker.scala
Thanks,
- Jon
On Oct 18, 8:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to
explain
the reason why this didn't happen, what we are doing about it, and what
the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs.
These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is
composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages
such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString()
in
your browsers console or by running the following JSON snippet through
your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and
will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to include a
string version of any ID when responding in the JSON format. What this
means
is Status, User, Direct Message and Saved Search IDs in the Twitter API
will
now be returned as an integer and a string in JSON responses. This will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str. The
following JSON representation of a status object shows the two versions
of
the ID fields for each data point.
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524
in_reply_to_status_id_str: 12738040524
}
]
What should you do - RIGHT NOW
--
The first thing you should do is attempt to decode the JSON snippet above
using your production code parser. Observe the output to confirm the ID
has
not lost accuracy.
What you do next depends on what happens:
* If your code converts the ID successfully without losing accuracy you
are
OK but should consider converting to the _str versions of IDs as soon as
possible.
* If your code has lost accuracy, convert your code to using the _str
version immediately. If you do not do this your code will be unable to
interact with the Twitter API reliably.
* In some language parsers, the JSON may throw an exception when reading
the
ID value. If this happens in your parser you will need to ‘pre-parse’ the
data, removing or replacing ID parameters with their _str versions.
Summary
-
1) If you develop in Javascript, know that you will have to update your
code
to read the string version instead of the integer version.
2) If you use a JSON decoder, validate that the example JSON, above,
decodes
without throwing exceptions. If exceptions are thrown, you will need to
pre-parse the data. Please let us know the name, version, and language of
the parser which throws the exception so we can investigate.
Timeline
---
by 22nd October 2010 (Friday): String versions of ID numbers will start
appearing in the API responses
4th November 2010 (Thursday) : Snowflake will be turned on but at ~41bit
Re: [twitter-dev] Re: Snowflake: An update and some very important information
I did some investigation into the snowflake algorithm recently and yes, it's
safe for 64bit signed longs. Even if Twitter moved away from using
scala/java longs internally (which are definitely signed), you'd still have
something like 65 years from now before the algorithm rolled past the 2^63-1
barrier.
I've posted a a gist[1] in ruby with a few little methods for playing with
the time part of a snowflake id if you're interested.
Hayes
1 http://gist.github.com/634586
On Tue, Oct 19, 2010 at 6:27 AM, Dan Checkoway dchecko...@gmail.com wrote:
I'm also patiently awaiting a response from twitter about this. Are the
ids sane for 64-bit *signed* long?
Dan
On Mon, Oct 18, 2010 at 9:08 PM, jon jonhoff...@gmail.com wrote:
Hi,
You wrote that the IDs are unsigned 64 bit ints, but the IdWorker is
pumping out java Longs which are signed. I'm assuming that was a
typo, but please clarify.
http://github.com/twitter/snowflake/blob/master/src/main/scala/com/twitter/service/snowflake/IdWorker.scala
Thanks,
- Jon
On Oct 18, 8:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to
explain
the reason why this didn't happen, what we are doing about it, and what
the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs.
These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is
composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages
such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString()
in
your browsers console or by running the following JSON snippet through
your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and
will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to include
a
string version of any ID when responding in the JSON format. What this
means
is Status, User, Direct Message and Saved Search IDs in the Twitter API
will
now be returned as an integer and a string in JSON responses. This will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str. The
following JSON representation of a status object shows the two versions
of
the ID fields for each data point.
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524
in_reply_to_status_id_str: 12738040524
}
]
What should you do - RIGHT NOW
--
The first thing you should do is attempt to decode the JSON snippet
above
using your production code parser. Observe the output to confirm the ID
has
not lost accuracy.
What you do next depends on what happens:
* If your code converts the ID successfully without losing accuracy you
are
OK but should consider converting to the _str versions of IDs as soon as
possible.
* If your code has lost accuracy, convert your code to using the _str
version immediately. If you do not do this your code will be unable to
interact with the Twitter API reliably.
* In some language parsers, the JSON may throw an exception when reading
the
ID value. If this happens in your parser you will need to ‘pre-parse’
the
data, removing or replacing ID parameters with their _str versions.
Summary
-
1) If you develop in Javascript, know that you will have to update your
code
to
Re: [twitter-dev] Re: Snowflake: An update and some very important information
I wouldn't blame this on JSON, because it's not JSON that has the
problems, but JavaScript. All of my Objective-C apps that communicate
use JSON as well, and they don't have the limitation. The issue does not
apply to XML either - there's no type specification in XML.
As far as I know, this issue will only cause trouble for a few
applications that work with JavaScript and depend on the IDs a lot.
My suggestion to solve this issue would be to introduce an additional
parameter (just like include_rts, just with a different name) that turns
all IDs into strings. No extra fields, just an additional optional
parameter. Won't cause trouble for the applications that can't parse it
and requires minimal implementation effort for developers.
I hope I'm not too late with my suggestion :-)
Tom
On 10/19/10 7:10 PM, Craig Hockenberry wrote:
This approach feels wrong to me. The red flag is the duplication of
data within the payload: in 30+ years of professional development,
I've never seen that work out well.
The root of the problem is that you've chosen to deliver data in a
format (JSON) that can't support integers with a value greater than
2^53 bits. And some of your data uses 2^64 bits.
The result is that you're working around the problem in a language by
using a string. Avoiding the root problem will encumber you with
legacy that you'll regret later.
Look at your proposed solution from a different point-of-view: say you
have a language that can't handle Unicode well (e.g. BASIC or Ruby.)
Would you solve this problem by adding another field called
text_ascii?
text: @themattharris hey how are things in København?.
text_ascii: @themattharris hey how are things in Kobenhavn?.
Seems silly, yet that is exactly what you're doing for Javascript and
long integers.
A part of this legacy in your payload is future confusion for
developers. Someone new to the Twitter API is going to be confused as
to why your ID values have both numeric and string representations.
And smart developers are going to lean towards the numeric
representation:
* 8 bytes of storage for 10765432100123456789 instead of 20 bytes.
* Faster sorting (less data to compare.)
* Correct sorting: 011 and 10 have different order depending on
whether you're sorting the string or numeric representation.
They'll eventually pay the price for choosing incorrectly.
Every ID in the API is going to need documentation as a result. For
example, are place IDs affected by this change? And what about the IDs
returned by the Search API? (there's no mention of since_id_str and
max_id_str above.)
Losing consistency with the XML format is also a problem. Unless
you're planning on adding _str elements to the XML payload, you're
presenting developers with a one-way street. A consumer of JSON
id_str can't easily change the format of data they want to consume.
In my mind, you really only have two good choices at this point:
1) Limit Snowflake's ID space to 2^53 bits. Easier for developers,
harder for Twitter.
2) Make all Twitter IDs into strings. Easier for Twitter, harder for
developers.
The second choice is obviously more disruptive, but if you really need
the ID space, it's the right one. Even if it means I need to make
major changes to my code.
On Oct 18, 5:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to explain
the reason why this didn't happen, what we are doing about it, and what the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs. These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString() in
your browsers console or by running the following JSON snippet through your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to include a
string version of any ID when responding in the JSON format. What this means
is Status, User, Direct Message and Saved Search IDs in the Twitter API will
now be returned as an integer and a string in JSON responses. This will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str. The
Re: [twitter-dev] Re: Snowflake: An update and some very important information
With all due respect, the root of the problem is that computer
scientists think in terms of abstract machines with infinitely-wide
registers, infinitely many addressable RAM cells, etc., and business
people think in terms of human populations and their tweet rates
growing geometrically for all time. Journalists believe neither of
these. ;-) And neither assumption is realistic, which is why we have
to make decisions like this from time to time, and why sometimes we
predict disasters like Y2K or 32-bit machines crumbling in 2038 that
don't actually happen. ;-)
So - for Twitter: what is your *realistic* projection for when a
53-bit integer ID will overflow? What are the underlying assumptions
about human population growth, spread of Twitter, revenue models,
competition, etc.? I know this is all highly confidential, so for sake
of argument, assume current tweet rates per user and the goal your
executives have stated of a billion users, with a plateau at that
point. The question I'm asking is whether you *really* need 64-bit
integer IDs for tweets or for users. ;-)
By the way, I ask similar questions of all the big data geeks out
there - so many naked emperors, so little time. ;-)
--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb
A mathematician is a device for turning coffee into theorems. - Paul Erdos
Quoting Craig Hockenberry craig.hockenbe...@gmail.com:
This approach feels wrong to me. The red flag is the duplication of
data within the payload: in 30+ years of professional development,
I've never seen that work out well.
The root of the problem is that you've chosen to deliver data in a
format (JSON) that can't support integers with a value greater than
2^53 bits. And some of your data uses 2^64 bits.
The result is that you're working around the problem in a language by
using a string. Avoiding the root problem will encumber you with
legacy that you'll regret later.
Look at your proposed solution from a different point-of-view: say you
have a language that can't handle Unicode well (e.g. BASIC or Ruby.)
Would you solve this problem by adding another field called
text_ascii?
text: @themattharris hey how are things in København?.
text_ascii: @themattharris hey how are things in Kobenhavn?.
Seems silly, yet that is exactly what you're doing for Javascript and
long integers.
A part of this legacy in your payload is future confusion for
developers. Someone new to the Twitter API is going to be confused as
to why your ID values have both numeric and string representations.
And smart developers are going to lean towards the numeric
representation:
* 8 bytes of storage for 10765432100123456789 instead of 20 bytes.
* Faster sorting (less data to compare.)
* Correct sorting: 011 and 10 have different order depending on
whether you're sorting the string or numeric representation.
They'll eventually pay the price for choosing incorrectly.
Every ID in the API is going to need documentation as a result. For
example, are place IDs affected by this change? And what about the IDs
returned by the Search API? (there's no mention of since_id_str and
max_id_str above.)
Losing consistency with the XML format is also a problem. Unless
you're planning on adding _str elements to the XML payload, you're
presenting developers with a one-way street. A consumer of JSON
id_str can't easily change the format of data they want to consume.
In my mind, you really only have two good choices at this point:
1) Limit Snowflake's ID space to 2^53 bits. Easier for developers,
harder for Twitter.
2) Make all Twitter IDs into strings. Easier for Twitter, harder for
developers.
The second choice is obviously more disruptive, but if you really need
the ID space, it's the right one. Even if it means I need to make
major changes to my code.
On Oct 18, 5:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to explain
the reason why this didn't happen, what we are doing about it, and what the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs. These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString() in
your browsers console or by running the following JSON snippet through your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and will
lose accuracy. In some
Re: [twitter-dev] Re: Snowflake: An update and some very important information
You normally don't need all 53 bits - but as snowflake simply skips a few IDs every now and then, you get a lot more IDs. Chance of it actually reaching 53 bits? I'd say that it happens at the end of November... Friday the 26th? Tom On 10/19/10 8:04 PM, M. Edward (Ed) Borasky wrote: With all due respect, the root of the problem is that computer scientists think in terms of abstract machines with infinitely-wide registers, infinitely many addressable RAM cells, etc., and business people think in terms of human populations and their tweet rates growing geometrically for all time. Journalists believe neither of these. ;-) And neither assumption is realistic, which is why we have to make decisions like this from time to time, and why sometimes we predict disasters like Y2K or 32-bit machines crumbling in 2038 that don't actually happen. ;-) So - for Twitter: what is your *realistic* projection for when a 53-bit integer ID will overflow? What are the underlying assumptions about human population growth, spread of Twitter, revenue models, competition, etc.? I know this is all highly confidential, so for sake of argument, assume current tweet rates per user and the goal your executives have stated of a billion users, with a plateau at that point. The question I'm asking is whether you *really* need 64-bit integer IDs for tweets or for users. ;-) By the way, I ask similar questions of all the big data geeks out there - so many naked emperors, so little time. ;-) -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
