Re: Sequence Number Generation With Zookeeper
Good news! I got approval to release this code! (Man, I love working for a startup!!!) :-) So anyone know: what's the next step? Do I need to obtain commit privileges? Or do I deliver the code to someone who has commit privs who shepherds this for me? Also, what (if anything) do I need to tweak in the code to make it release-ready. (e.g., Change package names? Slap an Apache license on it? etc.) Thanks, DR On 08/06/2010 10:39 PM, David Rosenstrauch wrote: I'll run it by my boss next week. DR On 08/06/2010 07:30 PM, Mahadev Konar wrote: Hi David, I think it would be really useful. It would be very helpful for someone looking for geenrating unique tokens/generations ids ( I can think of plenty of applications for this). Please do consider contributing it back to the community! Thanks mahadev On 8/6/10 7:10 AM, David Rosenstrauchdar...@darose.net wrote: Perhaps. I'd have to ask my boss for permission to release the code. Is this something that would be interesting/useful to other people? If so, I can ask about it. DR On 08/05/2010 11:02 PM, Jonathan Holloway wrote: Hi David, We did discuss potentially doing this as well. It would be nice to get some recipes for Zookeeper done for this area, if people think it's useful. Were you thinking of submitting this back as a recipe, if not then I could potentially work on such a recipe instead. Many thanks, Jon. I just ran into this exact situation, and handled it like so: I wrote a library that uses the option (b) you described above. Only instead of requesting a single sequence number, you request a block of them at a time from Zookeeper, and then locally use them up one by one from the block you retrieved. Retrieving by block (e.g., by blocks of 1 at a time) eliminates the contention issue. Then, if you're finished assigning ID's from that block, but still have a bunch of ID's left in the block, the library has another function to push back the unused ID's. They'll then get pulled again in the next block retrieval. We don't actually have this code running in production yet, so I can't vouch for how well it works. But the design was reviewed and given the thumbs up by the core developers on the team, and the implementation passes all my unit tests. HTH. Feel free to email back with specific questions if you'd like more details. DR
Re: Sequence Number Generation With Zookeeper
Great! Basic details are here (create a jira, attach a patch, click submit and someone will review and help you get it into a state which we can commit). Probably you'd put your code into src/recipes or src/contrib (recipes sounds reasonable). http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute Patrick On 08/10/2010 09:59 AM, David Rosenstrauch wrote: Good news! I got approval to release this code! (Man, I love working for a startup!!!) :-) So anyone know: what's the next step? Do I need to obtain commit privileges? Or do I deliver the code to someone who has commit privs who shepherds this for me? Also, what (if anything) do I need to tweak in the code to make it release-ready. (e.g., Change package names? Slap an Apache license on it? etc.) Thanks, DR On 08/06/2010 10:39 PM, David Rosenstrauch wrote: I'll run it by my boss next week. DR On 08/06/2010 07:30 PM, Mahadev Konar wrote: Hi David, I think it would be really useful. It would be very helpful for someone looking for geenrating unique tokens/generations ids ( I can think of plenty of applications for this). Please do consider contributing it back to the community! Thanks mahadev On 8/6/10 7:10 AM, David Rosenstrauchdar...@darose.net wrote: Perhaps. I'd have to ask my boss for permission to release the code. Is this something that would be interesting/useful to other people? If so, I can ask about it. DR On 08/05/2010 11:02 PM, Jonathan Holloway wrote: Hi David, We did discuss potentially doing this as well. It would be nice to get some recipes for Zookeeper done for this area, if people think it's useful. Were you thinking of submitting this back as a recipe, if not then I could potentially work on such a recipe instead. Many thanks, Jon. I just ran into this exact situation, and handled it like so: I wrote a library that uses the option (b) you described above. Only instead of requesting a single sequence number, you request a block of them at a time from Zookeeper, and then locally use them up one by one from the block you retrieved. Retrieving by block (e.g., by blocks of 1 at a time) eliminates the contention issue. Then, if you're finished assigning ID's from that block, but still have a bunch of ID's left in the block, the library has another function to push back the unused ID's. They'll then get pulled again in the next block retrieval. We don't actually have this code running in production yet, so I can't vouch for how well it works. But the design was reviewed and given the thumbs up by the core developers on the team, and the implementation passes all my unit tests. HTH. Feel free to email back with specific questions if you'd like more details. DR
Re: Sequence Number Generation With Zookeeper
OK, will do, as soon as time permits. It'll take me a little while to do the needed tweaks. (Plus I'm under some pretty heavy deadline on some other work right now.) Will email back once I've got this done. DR On 08/10/2010 01:10 PM, Patrick Hunt wrote: Great! Basic details are here (create a jira, attach a patch, click submit and someone will review and help you get it into a state which we can commit). Probably you'd put your code into src/recipes or src/contrib (recipes sounds reasonable). http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute Patrick On 08/10/2010 09:59 AM, David Rosenstrauch wrote: Good news! I got approval to release this code! (Man, I love working for a startup!!!) :-) So anyone know: what's the next step? Do I need to obtain commit privileges? Or do I deliver the code to someone who has commit privs who shepherds this for me? Also, what (if anything) do I need to tweak in the code to make it release-ready. (e.g., Change package names? Slap an Apache license on it? etc.) Thanks, DR