Re: Memcached Hackathon Report(s)
On Oct 25, 7:09 pm, dormando <[EMAIL PROTECTED]> wrote: > Well... I'm trying to think of the best way to do the merge right now. I tried that a few nights ago... it's not that clean. > It's time to pull the binary tree into the "canonical" repo and start > pushing changes/tags/releases there, but do we fix up the tree first? > > Dustin, if you have a preferred list of author contacts and wouldn't mind > me poking you for some hints, could you please send it over? I'll give a > shot at rewriting the tree. The easiest way is to probably redo the svn import with an authorfile and replay all of the branch changes with format-patch/am. filter-branch's most obvious use case ends up being rather hard. > Another option is ignoring it for now... I make a 'binary' branch based > off of everyone's latest changes, push that, tag it as 1.3.1 and ship. > Then rewrite the tree for 1.3.2? :/ Not sure how I feel about that, since > it could make it harder for people to hop on and help fix bugs in the > tree. Rebase is actually really smart about major tree rewrites, of course, it gets harder with merges and *large* rewrites. There's a lot of other stuff to do, of course. It'd also be good to have standards on commits before bringing them into trees. I've been a little bad about that
Re: Memcached Hackathon Report(s)
> I would actually like to do a rewrite of this branch to fix the > authors at some point. That will require coordination, but I think > it'd be valuable to make sure everyone is credited for the work that > they put into the project. Well... I'm trying to think of the best way to do the merge right now. It's time to pull the binary tree into the "canonical" repo and start pushing changes/tags/releases there, but do we fix up the tree first? Dustin, if you have a preferred list of author contacts and wouldn't mind me poking you for some hints, could you please send it over? I'll give a shot at rewriting the tree. Another option is ignoring it for now... I make a 'binary' branch based off of everyone's latest changes, push that, tag it as 1.3.1 and ship. Then rewrite the tree for 1.3.2? :/ Not sure how I feel about that, since it could make it harder for people to hop on and help fix bugs in the tree. Thoughts? -Dormando
Re: Memcached Hackathon Report(s)
\ On Oct 19, 9:58 am, Trond Norbye <[EMAIL PROTECTED]> wrote: > Hi, > > Eric Lambert fixed up some of the wording in the binary protocol > specification (check outhttp://github.com/elambert/memcached/tree/binprot > ) > > Please make sure we get this fix merged in as well :) This is all cleared up: http://github.com/dustin/memcached/commit/cf0b3151 Thanks everyone.
Re: Memcached Hackathon Report(s)
On Oct 19, 9:58 am, Trond Norbye <[EMAIL PROTECTED]> wrote: > Hi, > > Eric Lambert fixed up some of the wording in the binary protocol > specification (check outhttp://github.com/elambert/memcached/tree/binprot > ) > > Please make sure we get this fix merged in as well :) I'd like to make sure that contributors have git set up properly before sending in changes. Can you guys make sure that on every machine on which you're doing work, you issue the following commands: git config --global user.email [EMAIL PROTECTED] git config --global user.name 'Your Name' This is a really common thing that most people seem to miss and it prevents us from properly crediting the authors. This change may be quite good, but I wouldn't want it included in the tree as "[EMAIL PROTECTED]" I would actually like to do a rewrite of this branch to fix the authors at some point. That will require coordination, but I think it'd be valuable to make sure everyone is credited for the work that they put into the project. In the meantime, if Eric Lambert wants to get ahold of me directly ([EMAIL PROTECTED]), I can get this changeset rewritten and rebased as to ensure proper credit is given.
Re: Memcached Hackathon Report(s)
On Oct 19, 6:21 am, "Toru Maesaka" <[EMAIL PROTECTED]> wrote: > Hi guys, > > I fixed the stats opaque issue pointed out by Dustin's test case at > the hackathon and just pushed it to github: > > http://github.com/tmaesaka/memcached/commit/86159eb7b5c611d25a33eb1cb... Great, that's some code I can remove, then. :) > Oh, and I noticed that Dustin had merged my binary tree and there is > something I feel really guilty about... Some pushes I made to my tree > were done manually to bring over 1.2.6/1.2.7 stable patches into the > binary tree. This is probably going to hit us badly when merging other > trees even with the same ancestor (e.g. > brad'shttp://github.com/bradfitz/memcached/commits/incr_fix). > > Theres probably a way to fix this but I'm not quite a git ninja yet... > Does anyone have any thoughts/solution for this? Generally git will just do the right thing here. I just tried a merge against dormando/stable and it made a pretty big mess, though. Part of that may be because of the svn stuff that went on. At some point, we need to put it all together. I'm not going to worry about it too much, though. It will settle down fairly shortly and we can work on keeping the tree a bit cleaner after that merge.
Re: Memcached Hackathon Report(s)
On Oct 16, 2008, at 9:08 AM, steve.yen wrote: - For those of you who thought a hash table per thread might be interesting (like me), Trond says it's been tried and is not an improvement. The contention just moves towards the request queueing in front of each thread. I have not tested this, but I have "analyzed" it... The problem of using a "per thread" hashtable is that it is the clients and not the items that are bound to a thread. A client may request any object, so that the item you try to request may just as well belong to the pool belonging to another thread.. cheers Trond
Re: Memcached Hackathon Report(s)
On Oct 15, 2008, at 10:48 AM, Dustin wrote: The hackathon went well again. Admittedly, I didn't talk to everyone there, but from the corner with the food, there was some additional protocol work and tree merging going on. Please fill in interesting things that happened that I didn't know about or forgot. I would like to thank Matt Ingenthron for all the work arranging the hackathon! You did a fabulous job pulling all the strings making this a success! Thank you Matt, I really appreciate it! Cheers, Trond
Re: Memcached Hackathon Report(s)
Hi, Eric Lambert fixed up some of the wording in the binary protocol specification (check out http://github.com/elambert/memcached/tree/binprot ) Please make sure we get this fix merged in as well :) Cheers, Trond On Oct 19, 2008, at 3:21 PM, Toru Maesaka wrote: Hi guys, I fixed the stats opaque issue pointed out by Dustin's test case at the hackathon and just pushed it to github: http://github.com/tmaesaka/memcached/commit/86159eb7b5c611d25a33eb1cb6c750c806350912 I also refactored 'server_stats()' which is a function that returns server specific stats. In the previous implementation, a string in the usual stats format was created and returned if server_stats() was working for the ascii protocol. Otherwise, server_stats() would parse the string and create binary protocol packets based on that data. The above approach is not nice at all and I didn't like it from the beginning. It was implemented like that because it was written before the callback idea was adopted. Now there are no conditional statements to determine whether server_stats() is working for the binary protocol or not (because the appropriate callback is now passed to server_stats). Oh, and I noticed that Dustin had merged my binary tree and there is something I feel really guilty about... Some pushes I made to my tree were done manually to bring over 1.2.6/1.2.7 stable patches into the binary tree. This is probably going to hit us badly when merging other trees even with the same ancestor (e.g. brad's http://github.com/bradfitz/memcached/commits/incr_fix). Theres probably a way to fix this but I'm not quite a git ninja yet... Does anyone have any thoughts/solution for this? Cheers, Toru On Fri, Oct 17, 2008 at 5:08 AM, Ben Manes <[EMAIL PROTECTED]> wrote: "Trond also mentioned interesting lock reduction ideas, like moving from a single cache lock to lock per hash region (perhaps like java concurrent hash map). I think Marc, in a different conversation, suggested the idea of a lock-free hash table (ala Cliff Click)." You may also want to consider the hopscotch algorithm: http://groups.google.com/group/hopscotch-hashing?pli=1 - Original Message From: steve.yen <[EMAIL PROTECTED]> To: memcached Sent: Thursday, October 16, 2008 9:08:26 AM Subject: Re: Memcached Hackathon Report(s) A fun hackathon, folks were there until 2am. Other conversations that I recall... - Paul Saab & Marc Kwiatkowski described a bunch of the Facebook changes that helped them get towards 300K reqs/sec on 8 core server. I think they worked a bit towards putting up an svn/git repo so their changes could be more easily folded back by the community into the mainline (putting out something with change history rather than just a huge hard-to- digest patch). Appreciate that! - In particular, they described one performance blocker which they fixed of the single stats lock and how they moved towards a stats-per- thread model that tracks info in thread local data. These TLD stats are aggregated lazily, on-demand in response to a stats request. - Trond and I talked a little about making the upcoming storage engine interface async rather than synchronous flavored. Trond also mentioned interesting lock reduction ideas, like moving from a single cache lock to lock per hash region (perhaps like java concurrent hash map). I think Marc, in a different conversation, suggested the idea of a lock-free hash table (ala Cliff Click). - For those of you who thought a hash table per thread might be interesting (like me), Trond says it's been tried and is not an improvement. The contention just moves towards the request queueing in front of each thread. - There was talk and progress about adding replication into libmemcached client lib. Senthil (apologies if I misremembered your name) from Yahoo did the work, and we talked with Brian Aker about replication issues, such as handling flapping nodes. steve On Oct 15, 1:48 am, Dustin <[EMAIL PROTECTED]> wrote: The hackathon went well again. Admittedly, I didn't talk to everyone there, but from the corner with the food, there was some additional protocol work and tree merging going on. Please fill in interesting things that happened that I didn't know about or forgot. In particular, Toru implemented stats for the binary protocol. It's pretty good stuff. I wrote java client support for it, and helped iron out some minor issues that arose. I think the only change that came out of this was that the opaque behavior was kind of undefined. All opaque values coming out of stats were set to 0 (and the requested opaque was ignored). Stats values will have the magic opaque value of 0x746f7275 and the terminating stat will have the requested opaque value. *INCOMPATIBILITY NOTES* We have one change coming up that has been affecting me a lot re: compatibility and that's the removal of hold v
Re: Memcached Hackathon Report(s)
Hi guys, I fixed the stats opaque issue pointed out by Dustin's test case at the hackathon and just pushed it to github: http://github.com/tmaesaka/memcached/commit/86159eb7b5c611d25a33eb1cb6c750c806350912 I also refactored 'server_stats()' which is a function that returns server specific stats. In the previous implementation, a string in the usual stats format was created and returned if server_stats() was working for the ascii protocol. Otherwise, server_stats() would parse the string and create binary protocol packets based on that data. The above approach is not nice at all and I didn't like it from the beginning. It was implemented like that because it was written before the callback idea was adopted. Now there are no conditional statements to determine whether server_stats() is working for the binary protocol or not (because the appropriate callback is now passed to server_stats). Oh, and I noticed that Dustin had merged my binary tree and there is something I feel really guilty about... Some pushes I made to my tree were done manually to bring over 1.2.6/1.2.7 stable patches into the binary tree. This is probably going to hit us badly when merging other trees even with the same ancestor (e.g. brad's http://github.com/bradfitz/memcached/commits/incr_fix). Theres probably a way to fix this but I'm not quite a git ninja yet... Does anyone have any thoughts/solution for this? Cheers, Toru On Fri, Oct 17, 2008 at 5:08 AM, Ben Manes <[EMAIL PROTECTED]> wrote: > "Trond also mentioned interesting lock reduction ideas, like moving from a > single cache lock to lock per hash region (perhaps like java concurrent hash > map). I think Marc, in a different conversation, suggested the idea of a > lock-free hash table (ala Cliff Click)." > > You may also want to consider the hopscotch algorithm: > http://groups.google.com/group/hopscotch-hashing?pli=1 > > - Original Message > From: steve.yen <[EMAIL PROTECTED]> > To: memcached > Sent: Thursday, October 16, 2008 9:08:26 AM > Subject: Re: Memcached Hackathon Report(s) > > > A fun hackathon, folks were there until 2am. Other conversations that > I recall... > > - Paul Saab & Marc Kwiatkowski described a bunch of the Facebook > changes that helped them get towards 300K reqs/sec on 8 core server. > I think they worked a bit towards putting up an svn/git repo so their > changes > could be more easily folded back by the community into the mainline > (putting > out something with change history rather than just a huge hard-to- > digest patch). > Appreciate that! > > - In particular, they described one performance blocker which they > fixed > of the single stats lock and how they moved towards a stats-per- > thread > model that tracks info in thread local data. These TLD stats are > aggregated lazily, on-demand in response to a stats request. > > - Trond and I talked a little about making the upcoming storage engine > interface > async rather than synchronous flavored. Trond also mentioned > interesting lock > reduction ideas, like moving from a single cache lock to lock per hash > region > (perhaps like java concurrent hash map). I think Marc, in a different > conversation, > suggested the idea of a lock-free hash table (ala Cliff Click). > > - For those of you who thought a hash table per thread might be > interesting (like me), > Trond says it's been tried and is not an improvement. The contention > just moves > towards the request queueing in front of each thread. > > - There was talk and progress about adding replication into > libmemcached client lib. > Senthil (apologies if I misremembered your name) from Yahoo did the > work, and we > talked with Brian Aker about replication issues, such as handling > flapping nodes. > > steve > > On Oct 15, 1:48 am, Dustin <[EMAIL PROTECTED]> wrote: >> The hackathon went well again. Admittedly, I didn't talk to everyone >> there, but from the corner with the food, there was some additional >> protocol work and tree merging going on. Please fill in interesting >> things that happened that I didn't know about or forgot. >> >> In particular, Toru implemented stats for the binary protocol. It's >> pretty good stuff. I wrote java client support for it, and helped >> iron out some minor issues that arose. >> >> I think the only change that came out of this was that the opaque >> behavior was kind of undefined. All opaque values coming out of stats >> were set to 0 (and the requested opaque was ignored). Stats values >> will have the magic opaque value of 0x746f7275 and the terminating >> stat will have the requested opaque value. >> >> *INCOMPATIBILI
Re: Memcached Hackathon Report(s)
"Trond also mentioned interesting lock reduction ideas, like moving from a single cache lock to lock per hash region (perhaps like java concurrent hash map). I think Marc, in a different conversation, suggested the idea of a lock-free hash table (ala Cliff Click)." You may also want to consider the hopscotch algorithm: http://groups.google.com/group/hopscotch-hashing?pli=1 - Original Message From: steve.yen <[EMAIL PROTECTED]> To: memcached Sent: Thursday, October 16, 2008 9:08:26 AM Subject: Re: Memcached Hackathon Report(s) A fun hackathon, folks were there until 2am. Other conversations that I recall... - Paul Saab & Marc Kwiatkowski described a bunch of the Facebook changes that helped them get towards 300K reqs/sec on 8 core server. I think they worked a bit towards putting up an svn/git repo so their changes could be more easily folded back by the community into the mainline (putting out something with change history rather than just a huge hard-to- digest patch). Appreciate that! - In particular, they described one performance blocker which they fixed of the single stats lock and how they moved towards a stats-per- thread model that tracks info in thread local data. These TLD stats are aggregated lazily, on-demand in response to a stats request. - Trond and I talked a little about making the upcoming storage engine interface async rather than synchronous flavored. Trond also mentioned interesting lock reduction ideas, like moving from a single cache lock to lock per hash region (perhaps like java concurrent hash map). I think Marc, in a different conversation, suggested the idea of a lock-free hash table (ala Cliff Click). - For those of you who thought a hash table per thread might be interesting (like me), Trond says it's been tried and is not an improvement. The contention just moves towards the request queueing in front of each thread. - There was talk and progress about adding replication into libmemcached client lib. Senthil (apologies if I misremembered your name) from Yahoo did the work, and we talked with Brian Aker about replication issues, such as handling flapping nodes. steve On Oct 15, 1:48 am, Dustin <[EMAIL PROTECTED]> wrote: > The hackathon went well again. Admittedly, I didn't talk to everyone > there, but from the corner with the food, there was some additional > protocol work and tree merging going on. Please fill in interesting > things that happened that I didn't know about or forgot. > > In particular, Toru implemented stats for the binary protocol. It's > pretty good stuff. I wrote java client support for it, and helped > iron out some minor issues that arose. > > I think the only change that came out of this was that the opaque > behavior was kind of undefined. All opaque values coming out of stats > were set to 0 (and the requested opaque was ignored). Stats values > will have the magic opaque value of 0x746f7275 and the terminating > stat will have the requested opaque value. > > *INCOMPATIBILITY NOTES* > > We have one change coming up that has been affecting me a lot re: > compatibility and that's the removal of hold values from delete. If > you rely on this behavior, please let someone know -- it's going away. > > Brad did some work on making the long parser more strict in incr/ > decr. If you rely on arbitrary objects being treated as 0, you have a > silent bug that will turn into a loud bug in the future.
Re: Memcached Hackathon Report(s)
A fun hackathon, folks were there until 2am. Other conversations that I recall... - Paul Saab & Marc Kwiatkowski described a bunch of the Facebook changes that helped them get towards 300K reqs/sec on 8 core server. I think they worked a bit towards putting up an svn/git repo so their changes could be more easily folded back by the community into the mainline (putting out something with change history rather than just a huge hard-to- digest patch). Appreciate that! - In particular, they described one performance blocker which they fixed of the single stats lock and how they moved towards a stats-per- thread model that tracks info in thread local data. These TLD stats are aggregated lazily, on-demand in response to a stats request. - Trond and I talked a little about making the upcoming storage engine interface async rather than synchronous flavored. Trond also mentioned interesting lock reduction ideas, like moving from a single cache lock to lock per hash region (perhaps like java concurrent hash map). I think Marc, in a different conversation, suggested the idea of a lock-free hash table (ala Cliff Click). - For those of you who thought a hash table per thread might be interesting (like me), Trond says it's been tried and is not an improvement. The contention just moves towards the request queueing in front of each thread. - There was talk and progress about adding replication into libmemcached client lib. Senthil (apologies if I misremembered your name) from Yahoo did the work, and we talked with Brian Aker about replication issues, such as handling flapping nodes. steve On Oct 15, 1:48 am, Dustin <[EMAIL PROTECTED]> wrote: > The hackathon went well again. Admittedly, I didn't talk to everyone > there, but from the corner with the food, there was some additional > protocol work and tree merging going on. Please fill in interesting > things that happened that I didn't know about or forgot. > > In particular, Toru implemented stats for the binary protocol. It's > pretty good stuff. I wrote java client support for it, and helped > iron out some minor issues that arose. > > I think the only change that came out of this was that the opaque > behavior was kind of undefined. All opaque values coming out of stats > were set to 0 (and the requested opaque was ignored). Stats values > will have the magic opaque value of 0x746f7275 and the terminating > stat will have the requested opaque value. > > *INCOMPATIBILITY NOTES* > > We have one change coming up that has been affecting me a lot re: > compatibility and that's the removal of hold values from delete. If > you rely on this behavior, please let someone know -- it's going away. > > Brad did some work on making the long parser more strict in incr/ > decr. If you rely on arbitrary objects being treated as 0, you have a > silent bug that will turn into a loud bug in the future.
Re: Memcached Hackathon Report(s)
On Oct 15, 8:53 am, "Toru Maesaka" <[EMAIL PROTECTED]> wrote: > Hi, > > About the opaque issue in the stats implementation, as much as I love > the 0x746f7275 idea (it would be funny) > Trond came up with a very flexible solution that looks like the way to > go (this happened at like 1am). There goes your chance to be famous. > So, as far as the binary protocol spec goes, the server should really > send back what the client had supplied as opaque (rather than a magic > number), using a magic number would mean making an exception in the > protocol which is not so nice. OK, that's fine. In the meantime, my client will take any of the three values we've seen and/or discussed. I'll undo that at some point. > - We don't have to make an exception in the protocol. > - Since we are using a pointer, we can change the structure to > anything in the future. It does sound good. Also, note that the issues I was having with my deletion tests were because the hold was dropped so the server was refusing my hold request. All that's happy and I merged your binprot tree onto my binary tree. I actually would like to rewrite that branch for cleanliness. There are six different Tronds, two Torus, etc... I'd like to keep submissions a bit cleaner before they go upstream if possible. > Some client-side replication work was also being done for libmemcahed > as well I think. I remember Brian mentioning that they got a lot done > last night. I still haven't looked at client-side replication. It's one of those things that seems wrong, but people ask for it anyway. Thanks to Matt/Sun for hosting it and everybody else for showing up.
Re: Memcached Hackathon Report(s)
Oh, and I've shared the photos I've taken last night on Flickr too: http://www.flickr.com/photos/tmaesaka/sets/72157608050620544/ On Thu, Oct 16, 2008 at 12:53 AM, Toru Maesaka <[EMAIL PROTECTED]> wrote: > Hi, > > About the opaque issue in the stats implementation, as much as I love > the 0x746f7275 idea (it would be funny) > Trond came up with a very flexible solution that looks like the way to > go (this happened at like 1am). > > So, as far as the binary protocol spec goes, the server should really > send back what the client had supplied as opaque (rather than a magic > number), using a magic number would mean making an exception in the > protocol which is not so nice. > > The solution is to pass the engine a pointer to the connection > structure with the callback. The engine itself does not temper with > this pointer at all. All it does is, it feed the pointer to the > callback function, meaning the engine does not have to understand the > connection structure. Once the pointer is passed to the callback, the > callback understands the structure since it lives inside the core > server. > > So in this context, this pointer acts as a cookie and heres what we > can benefit from it: > > - We don't have to make an exception in the protocol. > - Since we are using a pointer, we can change the structure to > anything in the future. > > As for the client stuff, Jonathan went through and reviewed my binary > protocol patch for Cache::Memcached and found that some assumptions I > made in the code can be improved (protocol negotiation related). He is > also looking at optimizing the code for us by subclassing the patch > (reduces the number of conditional selections and hash lookups). So > the Perl folks will hopefully be able to enjoy the binary protocol in > no time :) > > Some client-side replication work was also being done for libmemcahed > as well I think. I remember Brian mentioning that they got a lot done > last night. > > Thats all I can remember for now but I'll post to this thread if I > remember anything else :) > > Cheers, > Toru > > > On Wed, Oct 15, 2008 at 5:48 PM, Dustin <[EMAIL PROTECTED]> wrote: >> >> The hackathon went well again. Admittedly, I didn't talk to everyone >> there, but from the corner with the food, there was some additional >> protocol work and tree merging going on. Please fill in interesting >> things that happened that I didn't know about or forgot. >> >> In particular, Toru implemented stats for the binary protocol. It's >> pretty good stuff. I wrote java client support for it, and helped >> iron out some minor issues that arose. >> >> I think the only change that came out of this was that the opaque >> behavior was kind of undefined. All opaque values coming out of stats >> were set to 0 (and the requested opaque was ignored). Stats values >> will have the magic opaque value of 0x746f7275 and the terminating >> stat will have the requested opaque value. >> >> >> *INCOMPATIBILITY NOTES* >> >> We have one change coming up that has been affecting me a lot re: >> compatibility and that's the removal of hold values from delete. If >> you rely on this behavior, please let someone know -- it's going away. >> >> Brad did some work on making the long parser more strict in incr/ >> decr. If you rely on arbitrary objects being treated as 0, you have a >> silent bug that will turn into a loud bug in the future. >> >
Re: Memcached Hackathon Report(s)
Hi, About the opaque issue in the stats implementation, as much as I love the 0x746f7275 idea (it would be funny) Trond came up with a very flexible solution that looks like the way to go (this happened at like 1am). So, as far as the binary protocol spec goes, the server should really send back what the client had supplied as opaque (rather than a magic number), using a magic number would mean making an exception in the protocol which is not so nice. The solution is to pass the engine a pointer to the connection structure with the callback. The engine itself does not temper with this pointer at all. All it does is, it feed the pointer to the callback function, meaning the engine does not have to understand the connection structure. Once the pointer is passed to the callback, the callback understands the structure since it lives inside the core server. So in this context, this pointer acts as a cookie and heres what we can benefit from it: - We don't have to make an exception in the protocol. - Since we are using a pointer, we can change the structure to anything in the future. As for the client stuff, Jonathan went through and reviewed my binary protocol patch for Cache::Memcached and found that some assumptions I made in the code can be improved (protocol negotiation related). He is also looking at optimizing the code for us by subclassing the patch (reduces the number of conditional selections and hash lookups). So the Perl folks will hopefully be able to enjoy the binary protocol in no time :) Some client-side replication work was also being done for libmemcahed as well I think. I remember Brian mentioning that they got a lot done last night. Thats all I can remember for now but I'll post to this thread if I remember anything else :) Cheers, Toru On Wed, Oct 15, 2008 at 5:48 PM, Dustin <[EMAIL PROTECTED]> wrote: > > The hackathon went well again. Admittedly, I didn't talk to everyone > there, but from the corner with the food, there was some additional > protocol work and tree merging going on. Please fill in interesting > things that happened that I didn't know about or forgot. > > In particular, Toru implemented stats for the binary protocol. It's > pretty good stuff. I wrote java client support for it, and helped > iron out some minor issues that arose. > > I think the only change that came out of this was that the opaque > behavior was kind of undefined. All opaque values coming out of stats > were set to 0 (and the requested opaque was ignored). Stats values > will have the magic opaque value of 0x746f7275 and the terminating > stat will have the requested opaque value. > > > *INCOMPATIBILITY NOTES* > > We have one change coming up that has been affecting me a lot re: > compatibility and that's the removal of hold values from delete. If > you rely on this behavior, please let someone know -- it's going away. > > Brad did some work on making the long parser more strict in incr/ > decr. If you rely on arbitrary objects being treated as 0, you have a > silent bug that will turn into a loud bug in the future. >
