Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
Hi all, To generate new keys/ UserIds for new users on my application, I am thinking of using a simple synchronized counter that can keep track of the no. of users registered on my application and when a new user signs up, he can be allotted the next available id. Since Cassandra is eventually consistent, Is this advisable to implement with Cassandra, but then I could also use stronger consistency level like quorum or all for this purpose. Please let me know your thoughts and suggesttions.. Regards Asil
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
Unless you need your user identifiers to be sequential for some reason, I would save yourself the headache of this kind of complexity and just use UUIDs if you have to generate an identifier. On Feb 3, 2011, at 2:03 PM, Aklin_81 wrote: > Hi all, > To generate new keys/ UserIds for new users on my application, I am > thinking of using a simple synchronized counter that can keep track of > the no. of users registered on my application and when a new user > signs up, he can be allotted the next available id. > > Since Cassandra is eventually consistent, Is this advisable to > implement with Cassandra, but then I could also use stronger > consistency level like quorum or all for this purpose. > > > Please let me know your thoughts and suggesttions.. > > Regards > Asil
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
You could also consider snowflake: http://github.com/twitter/snowflake which gives you ids that roughly sort by time (but aren't sequential). -ryan On Thu, Feb 3, 2011 at 11:13 AM, Matthew E. Kennedy wrote: > Unless you need your user identifiers to be sequential for some reason, I > would save yourself the headache of this kind of complexity and just use > UUIDs if you have to generate an identifier. > > On Feb 3, 2011, at 2:03 PM, Aklin_81 wrote: > >> Hi all, >> To generate new keys/ UserIds for new users on my application, I am >> thinking of using a simple synchronized counter that can keep track of >> the no. of users registered on my application and when a new user >> signs up, he can be allotted the next available id. >> >> Since Cassandra is eventually consistent, Is this advisable to >> implement with Cassandra, but then I could also use stronger >> consistency level like quorum or all for this purpose. >> >> >> Please let me know your thoughts and suggesttions.. >> >> Regards >> Asil > > -- @rk
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
Thanks Matthew & Ryan, The main inspiration behind me trying to generate Ids in sequential manner is to reduce the size of the userId, since I am using it for heavy denormalization. UUIDs are 16 bytes long, but I can also have a unique Id in just 4 bytes, and since this is just a one time process when the user signs-up, it makes sense to try cutting down the space requirements, if it is feasible "without any downsides"(!?). I am also using userIds to attach to Id of the other data of the user on my application. If I could reduce the userId size that I can also reduce the size of other Ids, I could drastically cut down the space requirements. [Sorry for this question is not directly related to cassandra but I think Cassandra factors here because of its tuneable consistency] Regards Asil On Fri, Feb 4, 2011 at 1:09 AM, Ryan King wrote: > You could also consider snowflake: > > http://github.com/twitter/snowflake > > which gives you ids that roughly sort by time (but aren't sequential). > > -ryan > > On Thu, Feb 3, 2011 at 11:13 AM, Matthew E. Kennedy > wrote: >> Unless you need your user identifiers to be sequential for some reason, I >> would save yourself the headache of this kind of complexity and just use >> UUIDs if you have to generate an identifier. >> >> On Feb 3, 2011, at 2:03 PM, Aklin_81 wrote: >> >>> Hi all, >>> To generate new keys/ UserIds for new users on my application, I am >>> thinking of using a simple synchronized counter that can keep track of >>> the no. of users registered on my application and when a new user >>> signs up, he can be allotted the next available id. >>> >>> Since Cassandra is eventually consistent, Is this advisable to >>> implement with Cassandra, but then I could also use stronger >>> consistency level like quorum or all for this purpose. >>> >>> >>> Please let me know your thoughts and suggesttions.. >>> >>> Regards >>> Asil >> >> > > > > -- > @rk >
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
On Thu, Feb 3, 2011 at 9:12 PM, Aklin_81 wrote: > Thanks Matthew & Ryan, > > The main inspiration behind me trying to generate Ids in sequential > manner is to reduce the size of the userId, since I am using it for > heavy denormalization. UUIDs are 16 bytes long, but I can also have a > unique Id in just 4 bytes, and since this is just a one time process > when the user signs-up, it makes sense to try cutting down the space > requirements, if it is feasible "without any downsides"(!?). > > I am also using userIds to attach to Id of the other data of the user > on my application. If I could reduce the userId size that I can also > reduce the size of other Ids, I could drastically cut down the space > requirements. > > > [Sorry for this question is not directly related to cassandra but I > think Cassandra factors here because of its tuneable consistency] Don't generate these ids in cassandra. Use something like snowflake, flickr's ticket servers [2] or zookeeper sequential nodes. -ryan 1. http://github.com/twitter/snowflake 2. http://code.flickr.com/blog/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
Thanks so much Ryan for the links; I'll definitely take them into consideration. Just another thought which came to my mind:- perhaps it may be beneficial to store(or duplicate) some of the data like the Login credentials & particularly userId to User's Name mapping, etc (which is very heavily read), in a fast MyISAM table. This could solve the problem of keys though auto-generated unique & sequential primary keys. I could use the same keys for Cassandra rows for that user. And also since Cassandra reads are relatively slow, it makes sense to store data like userId to Name mapping in MyISAM as this data would be required after almost all queries to the database. Regards -Asil On Fri, Feb 4, 2011 at 10:14 PM, Ryan King wrote: > On Thu, Feb 3, 2011 at 9:12 PM, Aklin_81 wrote: >> Thanks Matthew & Ryan, >> >> The main inspiration behind me trying to generate Ids in sequential >> manner is to reduce the size of the userId, since I am using it for >> heavy denormalization. UUIDs are 16 bytes long, but I can also have a >> unique Id in just 4 bytes, and since this is just a one time process >> when the user signs-up, it makes sense to try cutting down the space >> requirements, if it is feasible "without any downsides"(!?). >> >> I am also using userIds to attach to Id of the other data of the user >> on my application. If I could reduce the userId size that I can also >> reduce the size of other Ids, I could drastically cut down the space >> requirements. >> >> >> [Sorry for this question is not directly related to cassandra but I >> think Cassandra factors here because of its tuneable consistency] > > Don't generate these ids in cassandra. Use something like snowflake, > flickr's ticket servers [2] or zookeeper sequential nodes. > > -ryan > > > 1. http://github.com/twitter/snowflake > 2. > http://code.flickr.com/blog/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ >
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
If you mix mysql and Cassandra you risk creating a single point of failure around the mysql system. If you have use data that changes infrequently, a row cache in cassandra will give you fast reads. Aaron On 5/02/2011, at 8:13 AM, Aklin_81 wrote: > Thanks so much Ryan for the links; I'll definitely take them into > consideration. > > Just another thought which came to my mind:- > perhaps it may be beneficial to store(or duplicate) some of the data > like the Login credentials & particularly userId to User's Name > mapping, etc (which is very heavily read), in a fast MyISAM table. > This could solve the problem of keys though auto-generated unique & > sequential primary keys. I could use the same keys for Cassandra rows > for that user. And also since Cassandra reads are relatively slow, it > makes sense to store data like userId to Name mapping in MyISAM as > this data would be required after almost all queries to the database. > > Regards > -Asil > > > > On Fri, Feb 4, 2011 at 10:14 PM, Ryan King wrote: >> On Thu, Feb 3, 2011 at 9:12 PM, Aklin_81 wrote: >>> Thanks Matthew & Ryan, >>> >>> The main inspiration behind me trying to generate Ids in sequential >>> manner is to reduce the size of the userId, since I am using it for >>> heavy denormalization. UUIDs are 16 bytes long, but I can also have a >>> unique Id in just 4 bytes, and since this is just a one time process >>> when the user signs-up, it makes sense to try cutting down the space >>> requirements, if it is feasible "without any downsides"(!?). >>> >>> I am also using userIds to attach to Id of the other data of the user >>> on my application. If I could reduce the userId size that I can also >>> reduce the size of other Ids, I could drastically cut down the space >>> requirements. >>> >>> >>> [Sorry for this question is not directly related to cassandra but I >>> think Cassandra factors here because of its tuneable consistency] >> >> Don't generate these ids in cassandra. Use something like snowflake, >> flickr's ticket servers [2] or zookeeper sequential nodes. >> >> -ryan >> >> >> 1. http://github.com/twitter/snowflake >> 2. >> http://code.flickr.com/blog/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ >>
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
Why not synchronize on the client side? Make sure that the process that allocates user ids runs on only a single machine, in a synchronized method, and uses QUORUM for its reads and writes to Cassandra? On Sun, Feb 6, 2011 at 11:02 PM, Aaron Morton wrote: > If you mix mysql and Cassandra you risk creating a single point of failure > around the mysql system. > > If you have use data that changes infrequently, a row cache in cassandra > will give you fast reads. > > Aaron > > On 5/02/2011, at 8:13 AM, Aklin_81 wrote: > > > Thanks so much Ryan for the links; I'll definitely take them into > > consideration. > > > > Just another thought which came to my mind:- > > perhaps it may be beneficial to store(or duplicate) some of the data > > like the Login credentials & particularly userId to User's Name > > mapping, etc (which is very heavily read), in a fast MyISAM table. > > This could solve the problem of keys though auto-generated unique & > > sequential primary keys. I could use the same keys for Cassandra rows > > for that user. And also since Cassandra reads are relatively slow, it > > makes sense to store data like userId to Name mapping in MyISAM as > > this data would be required after almost all queries to the database. > > > > Regards > > -Asil > > > > > > > > On Fri, Feb 4, 2011 at 10:14 PM, Ryan King wrote: > >> On Thu, Feb 3, 2011 at 9:12 PM, Aklin_81 wrote: > >>> Thanks Matthew & Ryan, > >>> > >>> The main inspiration behind me trying to generate Ids in sequential > >>> manner is to reduce the size of the userId, since I am using it for > >>> heavy denormalization. UUIDs are 16 bytes long, but I can also have a > >>> unique Id in just 4 bytes, and since this is just a one time process > >>> when the user signs-up, it makes sense to try cutting down the space > >>> requirements, if it is feasible "without any downsides"(!?). > >>> > >>> I am also using userIds to attach to Id of the other data of the user > >>> on my application. If I could reduce the userId size that I can also > >>> reduce the size of other Ids, I could drastically cut down the space > >>> requirements. > >>> > >>> > >>> [Sorry for this question is not directly related to cassandra but I > >>> think Cassandra factors here because of its tuneable consistency] > >> > >> Don't generate these ids in cassandra. Use something like snowflake, > >> flickr's ticket servers [2] or zookeeper sequential nodes. > >> > >> -ryan > >> > >> > >> 1. http://github.com/twitter/snowflake > >> 2. > http://code.flickr.com/blog/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/ > >> >
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
Hi Ryan, I am considering snowflake as an option for my usage with Cassandra for a distributed application. As I came to know snowflake uses 64 bits IDs. I am looking for a solution that could help me generate 64 bits Ids but in those 64 bits I would like at least 4 free bits so that I could manipulate with those free bits to distinguish the two rows for a same entity(split by kind of data) in same column family. If I could keep the snowflake's Id size to around 60 bits, that would be great for my use case. Is it possible to manipulate the bits safely to around 60 bits? Perhaps the microsecond precision is not required to that much depth for my use case. Any kind of suggestions would be appreciated. Best Regards Ertio Lew On Fri, Feb 4, 2011 at 1:09 AM, Ryan King wrote: > You could also consider snowflake: > > http://github.com/twitter/snowflake > > which gives you ids that roughly sort by time (but aren't sequential). > > -ryan > > On Thu, Feb 3, 2011 at 11:13 AM, Matthew E. Kennedy > wrote: >> Unless you need your user identifiers to be sequential for some reason, I >> would save yourself the headache of this kind of complexity and just use >> UUIDs if you have to generate an identifier. >> >> On Feb 3, 2011, at 2:03 PM, Aklin_81 wrote: >> >>> Hi all, >>> To generate new keys/ UserIds for new users on my application, I am >>> thinking of using a simple synchronized counter that can keep track of >>> the no. of users registered on my application and when a new user >>> signs up, he can be allotted the next available id. >>> >>> Since Cassandra is eventually consistent, Is this advisable to >>> implement with Cassandra, but then I could also use stronger >>> consistency level like quorum or all for this purpose. >>> >>> >>> Please let me know your thoughts and suggesttions.. >>> >>> Regards >>> Asil >> >> > > > > -- > @rk >
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
This is mostly from memory. But the last 12 ? (4096 decimal) bits are a counter for the number of id's generated in a particular millisecond for that server. You could use the high 4 bits in that range for your data type flags and the low 8 for the counter. Aaron On 1/03/2011, at 4:41 AM, Ertio Lew wrote: > Hi Ryan, > > I am considering snowflake as an option for my usage with Cassandra > for a distributed application. > As I came to know snowflake uses 64 bits IDs. I am looking for a > solution that could help me generate 64 bits Ids > but in those 64 bits I would like at least 4 free bits so that I could > manipulate with those free bits to distinguish the two rows for a same > entity(split by kind of data) in same column family. > > If I could keep the snowflake's Id size to around 60 bits, that would > be great for my use case. Is it possible to manipulate the bits safely > to around 60 bits? Perhaps the microsecond precision is not required > to that much depth for my use case. > > Any kind of suggestions would be appreciated. > > Best Regards > Ertio Lew > > > > > > > > On Fri, Feb 4, 2011 at 1:09 AM, Ryan King wrote: >> You could also consider snowflake: >> >> http://github.com/twitter/snowflake >> >> which gives you ids that roughly sort by time (but aren't sequential). >> >> -ryan >> >> On Thu, Feb 3, 2011 at 11:13 AM, Matthew E. Kennedy >> wrote: >>> Unless you need your user identifiers to be sequential for some reason, I >>> would save yourself the headache of this kind of complexity and just use >>> UUIDs if you have to generate an identifier. >>> >>> On Feb 3, 2011, at 2:03 PM, Aklin_81 wrote: >>> Hi all, To generate new keys/ UserIds for new users on my application, I am thinking of using a simple synchronized counter that can keep track of the no. of users registered on my application and when a new user signs up, he can be allotted the next available id. Since Cassandra is eventually consistent, Is this advisable to implement with Cassandra, but then I could also use stronger consistency level like quorum or all for this purpose. Please let me know your thoughts and suggesttions.. Regards Asil >>> >>> >> >> >> >> -- >> @rk >>
Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up
On Tue, Mar 1, 2011 at 1:26 AM, Aaron Morton wrote: > This is mostly from memory. But the last 12 ? (4096 decimal) bits are a > counter for the number of id's generated in a particular millisecond for that > server. You could use the high 4 bits in that range for your data type flags > and the low 8 for the counter. So then I would be able to generate a maximum of upto 256 Ids per millisecond (or 256000 per second) on one machine!? Seems like a very good limit for my use case. I dont think I would ever need beyond that since my write volumes are quite below as compared to that limit.. Should I go for it or still are there any other things to consider ? > > Aaron > > On 1/03/2011, at 4:41 AM, Ertio Lew wrote: > >> Hi Ryan, >> >> I am considering snowflake as an option for my usage with Cassandra >> for a distributed application. >> As I came to know snowflake uses 64 bits IDs. I am looking for a >> solution that could help me generate 64 bits Ids >> but in those 64 bits I would like at least 4 free bits so that I could >> manipulate with those free bits to distinguish the two rows for a same >> entity(split by kind of data) in same column family. >> >> If I could keep the snowflake's Id size to around 60 bits, that would >> be great for my use case. Is it possible to manipulate the bits safely >> to around 60 bits? Perhaps the microsecond precision is not required >> to that much depth for my use case. >> >> Any kind of suggestions would be appreciated. >> >> Best Regards >> Ertio Lew >> >> >> >> >> >> >> >> On Fri, Feb 4, 2011 at 1:09 AM, Ryan King wrote: >>> You could also consider snowflake: >>> >>> http://github.com/twitter/snowflake >>> >>> which gives you ids that roughly sort by time (but aren't sequential). >>> >>> -ryan >>> >>> On Thu, Feb 3, 2011 at 11:13 AM, Matthew E. Kennedy >>> wrote: Unless you need your user identifiers to be sequential for some reason, I would save yourself the headache of this kind of complexity and just use UUIDs if you have to generate an identifier. On Feb 3, 2011, at 2:03 PM, Aklin_81 wrote: > Hi all, > To generate new keys/ UserIds for new users on my application, I am > thinking of using a simple synchronized counter that can keep track of > the no. of users registered on my application and when a new user > signs up, he can be allotted the next available id. > > Since Cassandra is eventually consistent, Is this advisable to > implement with Cassandra, but then I could also use stronger > consistency level like quorum or all for this purpose. > > > Please let me know your thoughts and suggesttions.. > > Regards > Asil >>> >>> >>> >>> -- >>> @rk >>> >
