Re: Facebook-memcached
On Jul 28, 11:56 am, Harish Kumar wrote: > Is there any one who could help me ? > At least few links which gives little more information. Is it a wrong > place to ask for this version of memcached ? I don't think anyone supports that version. We kind of look at it as a research project. Facebook does some good things, and it's nice of them to share some of the things that they've built, but they're not in the business of maintaining or supporting their fork of memcached. You've kind of come around here and asked for help using a piece of software to replace memcached, but without giving us the opportunity to fill the need you're perceiving. Are you certain your usage patterns can't be served by the community version? You mentioned fragmentation, but that really doesn't apply to memcached as it exists today. There are some bad conditions you can find yourself in, but I'd at least like to know what kind of actual bad experiences you're having so we can hopefully do something to make it better for others who find themselves in your situation (assuming we can't serve your own needs better).
Re: Facebook-memcached
Are you having a problem with performance, fragmentation and stuff in the current memcached version? On Tue, Jul 28, 2009 at 11:56 AM, Harish Kumar wrote: > > Is there any one who could help me ? > At least few links which gives little more information. Is it a wrong > place to ask for this version of memcached ? > > > On Jul 21, 12:59 pm, Harish Kumar wrote: >> Yes, there is an idea in our group which could be helpful in >> delivering the better performance and more sophisticated which reduces >> the fragmentation and stuff. So, I am interested in knowing about the >> code offacebook's memcachedto know how they are implementing so as >> to link it with the idea we had with the facebook's idea. >> >> Or can you give me a link where I can know more details of the code. >> >> On Jul 20, 11:52 pm, Dustin wrote: >> >> > On Jul 20, 2:07 pm, Harish Kumar wrote: >> >> > > I am kinda new to a research group and I am trying to understand the >> > > code offacebook-memcached. >> >> > You're asking questions about a somewhat uncommon piece of >> > software. Is there a particular reason you're looking at that version >> > instead of the community version? -- Eric Bergen eric.ber...@gmail.com http://www.ebergen.net
Re: Facebook-memcached
Is there any one who could help me ? At least few links which gives little more information. Is it a wrong place to ask for this version of memcached ? On Jul 21, 12:59 pm, Harish Kumar wrote: > Yes, there is an idea in our group which could be helpful in > delivering the better performance and more sophisticated which reduces > the fragmentation and stuff. So, I am interested in knowing about the > code offacebook's memcachedto know how they are implementing so as > to link it with the idea we had with the facebook's idea. > > Or can you give me a link where I can know more details of the code. > > On Jul 20, 11:52 pm, Dustin wrote: > > > On Jul 20, 2:07 pm, Harish Kumar wrote: > > > > I am kinda new to a research group and I am trying to understand the > > > code offacebook-memcached. > > > You're asking questions about a somewhat uncommon piece of > > software. Is there a particular reason you're looking at that version > > instead of the community version?
Re: Facebook-memcached
Yes, there is an idea in our group which could be helpful in delivering the better performance and more sophisticated which reduces the fragmentation and stuff. So, I am interested in knowing about the code of facebook's memcached to know how they are implementing so as to link it with the idea we had with the facebook's idea. Or can you give me a link where I can know more details of the code. On Jul 20, 11:52 pm, Dustin wrote: > On Jul 20, 2:07 pm, Harish Kumar wrote: > > > I am kinda new to a research group and I am trying to understand the > > code of facebook-memcached. > > You're asking questions about a somewhat uncommon piece of > software. Is there a particular reason you're looking at that version > instead of the community version?
Re: Facebook-memcached
On Jul 20, 2:07 pm, Harish Kumar wrote: > I am kinda new to a research group and I am trying to understand the > code of facebook-memcached. You're asking questions about a somewhat uncommon piece of software. Is there a particular reason you're looking at that version instead of the community version?
Re: facebook memcached on github
On Jan 9, 4:52 am, Arjen van der Meijden wrote: > On 9-1-2009 5:22 Du Song wrote: > > > It's already there. > > in PECL-memcache 3.x > >http://pecl.php.net/package/memcache > > > Regards, > > Du Song > > That code is still beta, which afaik it should be, since at least we > repeatedly run into segfaults in php, which dissappear when using the > stable 2.2-version. > Unfortunately we can't produce a reliable test-case that also segfaults > on the developer's systems. > > So if you have complex object-structures in your memcached, especially > when they do some work in the __wakeup(), you should be very careful > with choosing the 3.x-versions. Or hope you find another test case for > the same bug...http://pecl.php.net/bugs/bug.php?id=13623 > > Best regards, > > Arjen I see; I'll check this 3.0.2b and see how it goes. On another hand, by any chance, would it be possible for you Facebook guys to explain better the changes you guys made on the kernel to optimize it for UDP ? Diffs would be perfect =) I'm in the process of getting some machines up to use exclusively with memcached (8x2.5GHz w/ 64GB RAM) and it would be great if I could poke around with the optimizations you guys made (based on what I read on http://www.facebook.com/note.php?note_id=39391378919) Thanks!
Re: facebook memcached on github
On 9-1-2009 5:22 Du Song wrote: It's already there. in PECL-memcache 3.x http://pecl.php.net/package/memcache Regards, Du Song That code is still beta, which afaik it should be, since at least we repeatedly run into segfaults in php, which dissappear when using the stable 2.2-version. Unfortunately we can't produce a reliable test-case that also segfaults on the developer's systems. So if you have complex object-structures in your memcached, especially when they do some work in the __wakeup(), you should be very careful with choosing the 3.x-versions. Or hope you find another test case for the same bug... http://pecl.php.net/bugs/bug.php?id=13623 Best regards, Arjen
Re: facebook memcached on github
On Sun, Jan 4, 2009 at 23:43, Advt wrote: > > > > On Dec 13 2008, 11:24 am, marc kwiatkowski > wrote: >> As some of you may have noticed we've putfacebook'sversion of >> memcached up on github >> >> http://github.com/fbmarc/facebook-memcached >> >> We've done this for a couple of reasons. >> >> First, as far as we know, we run the largest installation of memcached >> in the world. We think this is true in terms of ops/secs, number of >> objects, amount of RAM, and number of servers. To get memcached to >> support these levels we've had to make a lot of customizations. We >> believe the memcache community would appreciate seeing what we've >> done. >> >> Second, a less complete repository of our version of memcached was >> already up on github. We thought it would be better to have a >> repository with our complete revision history both for our production >> version and our attempt at integration with the 1.2.5 release. >> >> As far as licensing goes. memcached was released under the BSD >> license. We cannot change that even if we wanted to. It goes without >> saying that we don't want to. >> >> Finally, I produced these branches by running git-svn and git-filter- >> branch against our internal SVN repositories. The addition of the >> root "src" directory and the lack of a clear branch from 1.2.3 make >> comparison with existing branches unnecessarily difficult. I >> apologize for the inconvenience. I will redo the git svn extraction >> to correct those issues later this week. >> >> In the mean time I hope you will let us know what you think and what >> else we can do to make memcached even better. We atfacebookwould be >> happy to discuss questions and issues about our release in this group. >> >> Thanks > > > Hi Marc, > > Thanks for making this public and congratz for the wonderful work ! > Any idea on when will the PHP client that uses UDP be available, if > ever ? > > Thanks, > Daniel It's already there. in PECL-memcache 3.x http://pecl.php.net/package/memcache Regards, Du Song
Re: facebook memcached on github
On Dec 13 2008, 11:24 am, marc kwiatkowski wrote: > As some of you may have noticed we've putfacebook'sversion of > memcached up on github > > http://github.com/fbmarc/facebook-memcached > > We've done this for a couple of reasons. > > First, as far as we know, we run the largest installation of memcached > in the world. We think this is true in terms of ops/secs, number of > objects, amount of RAM, and number of servers. To get memcached to > support these levels we've had to make a lot of customizations. We > believe the memcache community would appreciate seeing what we've > done. > > Second, a less complete repository of our version of memcached was > already up on github. We thought it would be better to have a > repository with our complete revision history both for our production > version and our attempt at integration with the 1.2.5 release. > > As far as licensing goes. memcached was released under the BSD > license. We cannot change that even if we wanted to. It goes without > saying that we don't want to. > > Finally, I produced these branches by running git-svn and git-filter- > branch against our internal SVN repositories. The addition of the > root "src" directory and the lack of a clear branch from 1.2.3 make > comparison with existing branches unnecessarily difficult. I > apologize for the inconvenience. I will redo the git svn extraction > to correct those issues later this week. > > In the mean time I hope you will let us know what you think and what > else we can do to make memcached even better. We atfacebookwould be > happy to discuss questions and issues about our release in this group. > > Thanks Hi Marc, Thanks for making this public and congratz for the wonderful work ! Any idea on when will the PHP client that uses UDP be available, if ever ? Thanks, Daniel
Re: facebook memcached on github (redux)
On Dec 21, 12:45 am, Tony Tung wrote: > It could be a separate storage engine, but not as spec'ed. The external > storage engine specification does not allow for scatter-gather storage of the > key and data. Can you be more specific as to how the current interface fails you? What changes would be required for it to do what you need?
Re: facebook memcached on github (redux)
On 12/20/08 4:42 AM, "Toru Maesaka" wrote: Hi! I've quickly glanced through the repo and even though I think the idea of the flat allocator is neat and how it uses mmap (/me loves mmap), I think the flat allocator itself should belong outside the codebase. So, what I'm trying to say is, shouldn't this be a separate storage engine? Trond and I are currently experimenting on reshaping the codebase by changing the item structure to be a minimal and generic 32/64bit aligned structure, which the storage engine will handle. If you're interested I can write more about this. So what I'd like to know is, what exactly is it that you guys are concerned about dynamic linking at startup? If the performance loss (if we could even call it that) is going to become a huge issue for the memcached userbase, we could discuss/plan static linking too. Personally, I think working on things like improving memcached's space efficiency (more bang for buck) seem more important than the dynamic linking issue. Cheers, Toru Opinions? ASFAIK, It could be a separate storage engine, but not as spec'ed. The external storage engine specification does not allow for scatter-gather storage of the key and data. Thanks, Tony
Re: facebook memcached on github (redux)
On Dec 20, 12:56 pm, Matt Ingenthron wrote: > True, more Trond than I. I've been working on a method of testing, and > have a good workload generator. We're actually not using mmap() at the > moment but we'd talked about it before. I don't think it would be a > complex change. > > Also, Trond had some thoughts on how to do this with less locking which > he'd sketched out, but hasn't told me about yet. I think he was working > on resynching everything with the latest engine code. Do any of these require things other than the engine abstraction? One of the main reasons for the engine abstraction was to allow for such experimentation, wasn't it?
Re: facebook memcached on github (redux)
Toru Maesaka wrote: oops, clicked send before I finished the email :( ASFAIK, Trond and Matt has been experimenting quite a bit with what I've mentioned above. True, more Trond than I. I've been working on a method of testing, and have a good workload generator. We're actually not using mmap() at the moment but we'd talked about it before. I don't think it would be a complex change. Also, Trond had some thoughts on how to do this with less locking which he'd sketched out, but hasn't told me about yet. I think he was working on resynching everything with the latest engine code. - Matt On Sat, Dec 20, 2008 at 9:42 PM, Toru Maesaka wrote: Hi! I've quickly glanced through the repo and even though I think the idea of the flat allocator is neat and how it uses mmap (/me loves mmap), I think the flat allocator itself should belong outside the codebase. So, what I'm trying to say is, shouldn't this be a separate storage engine? Trond and I are currently experimenting on reshaping the codebase by changing the item structure to be a minimal and generic 32/64bit aligned structure, which the storage engine will handle. If you're interested I can write more about this. So what I'd like to know is, what exactly is it that you guys are concerned about dynamic linking at startup? If the performance loss (if we could even call it that) is going to become a huge issue for the memcached userbase, we could discuss/plan static linking too. Personally, I think working on things like improving memcached's space efficiency (more bang for buck) seem more important than the dynamic linking issue. Cheers, Toru Opinions? ASFAIK,
Re: facebook memcached on github (redux)
oops, clicked send before I finished the email :( ASFAIK, Trond and Matt has been experimenting quite a bit with what I've mentioned above. Thanks, Toru On Sat, Dec 20, 2008 at 9:42 PM, Toru Maesaka wrote: > Hi! > > I've quickly glanced through the repo and even though I think the idea > of the flat allocator is neat and how it uses mmap (/me loves mmap), I > think the flat allocator itself should belong outside the codebase. > So, what I'm trying to say is, shouldn't this be a separate storage > engine? > > Trond and I are currently experimenting on reshaping the codebase by > changing the item structure to be a minimal and generic 32/64bit > aligned structure, which the storage engine will handle. If you're > interested I can write more about this. > > So what I'd like to know is, what exactly is it that you guys are > concerned about dynamic linking at startup? If the performance loss > (if we could even call it that) is going to become a huge issue for > the memcached userbase, we could discuss/plan static linking too. > > Personally, I think working on things like improving memcached's space > efficiency (more bang for buck) seem more important than the dynamic > linking issue. > > Cheers, > Toru > > Opinions? > ASFAIK, > > On Sat, Dec 20, 2008 at 4:28 AM, dormando wrote: >> >> As-is it doesn't seem to work very well. The slab must be empty, have no >> locked items, etc. I think brad also mentioned crashes in the code? I >> haven't tested it heavily myself, but the feature seems incomplete right >> now. >> >>> Do tell! :) >>> >>> On the protocol side: It'd be great to get it added to >>> http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt >>> and eventually to the various client libraries. >>> >>> On the implementation side: Does there need to be a completely stale >>> slab available? Does it / can it shuffle items around between slabs to >>> free one up (I presume not, since that wouldn't be O(1))? Does it / >>> can it LRU-out a bunch of items to free up a slab (also not O(1))? >>> Without some amount of item reshuffling / forced expiring, it's not >>> immediately clear to me how effective this would be. With the internal >>> reshuffling, I'd be worried about in-band performance on a heavily >>> used instance. Or perhaps I just don't know what I'm talking about. >>> >>> Seriously, though, it'd be super helpful to make this more visible. >>> Just to illustrate, my initial Googling on this topic brings up: >>> >>> (1) Complaints about the problem: http://www.linuxjournal.com/article/7451 >>> (2) Your client code: >>> http://github.com/dustin/java-memcached-client/tree/slab-reassign >>> (3) A hint that slab rebalancing is coming: >>> >>> http://code.sixapart.com/svn/memcached/trunk/server/doc/memory_management.txt >>> (4) Advice to bounce memcache: >>> http://www.socialtext.net/memcached/index.cgi?managing_a_memcached_cluster >>> >>> I'd be happy to help write up / clean up documentation...once I know >>> how it works. If you're swamped, I can just go read the code, but that >>> probably won't happen soon. :) >>> >>> Regards, >>> Josh >>> >> >
Re: facebook memcached on github (redux)
Hi! I've quickly glanced through the repo and even though I think the idea of the flat allocator is neat and how it uses mmap (/me loves mmap), I think the flat allocator itself should belong outside the codebase. So, what I'm trying to say is, shouldn't this be a separate storage engine? Trond and I are currently experimenting on reshaping the codebase by changing the item structure to be a minimal and generic 32/64bit aligned structure, which the storage engine will handle. If you're interested I can write more about this. So what I'd like to know is, what exactly is it that you guys are concerned about dynamic linking at startup? If the performance loss (if we could even call it that) is going to become a huge issue for the memcached userbase, we could discuss/plan static linking too. Personally, I think working on things like improving memcached's space efficiency (more bang for buck) seem more important than the dynamic linking issue. Cheers, Toru Opinions? ASFAIK, On Sat, Dec 20, 2008 at 4:28 AM, dormando wrote: > > As-is it doesn't seem to work very well. The slab must be empty, have no > locked items, etc. I think brad also mentioned crashes in the code? I > haven't tested it heavily myself, but the feature seems incomplete right > now. > >> Do tell! :) >> >> On the protocol side: It'd be great to get it added to >> http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt >> and eventually to the various client libraries. >> >> On the implementation side: Does there need to be a completely stale >> slab available? Does it / can it shuffle items around between slabs to >> free one up (I presume not, since that wouldn't be O(1))? Does it / >> can it LRU-out a bunch of items to free up a slab (also not O(1))? >> Without some amount of item reshuffling / forced expiring, it's not >> immediately clear to me how effective this would be. With the internal >> reshuffling, I'd be worried about in-band performance on a heavily >> used instance. Or perhaps I just don't know what I'm talking about. >> >> Seriously, though, it'd be super helpful to make this more visible. >> Just to illustrate, my initial Googling on this topic brings up: >> >> (1) Complaints about the problem: http://www.linuxjournal.com/article/7451 >> (2) Your client code: >> http://github.com/dustin/java-memcached-client/tree/slab-reassign >> (3) A hint that slab rebalancing is coming: >> >> http://code.sixapart.com/svn/memcached/trunk/server/doc/memory_management.txt >> (4) Advice to bounce memcache: >> http://www.socialtext.net/memcached/index.cgi?managing_a_memcached_cluster >> >> I'd be happy to help write up / clean up documentation...once I know >> how it works. If you're swamped, I can just go read the code, but that >> probably won't happen soon. :) >> >> Regards, >> Josh >> >
Re: facebook memcached on github (redux)
As-is it doesn't seem to work very well. The slab must be empty, have no locked items, etc. I think brad also mentioned crashes in the code? I haven't tested it heavily myself, but the feature seems incomplete right now. Do tell! :) On the protocol side: It'd be great to get it added to http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt and eventually to the various client libraries. On the implementation side: Does there need to be a completely stale slab available? Does it / can it shuffle items around between slabs to free one up (I presume not, since that wouldn't be O(1))? Does it / can it LRU-out a bunch of items to free up a slab (also not O(1))? Without some amount of item reshuffling / forced expiring, it's not immediately clear to me how effective this would be. With the internal reshuffling, I'd be worried about in-band performance on a heavily used instance. Or perhaps I just don't know what I'm talking about. Seriously, though, it'd be super helpful to make this more visible. Just to illustrate, my initial Googling on this topic brings up: (1) Complaints about the problem: http://www.linuxjournal.com/article/7451 (2) Your client code: http://github.com/dustin/java-memcached-client/tree/slab-reassign (3) A hint that slab rebalancing is coming: http://code.sixapart.com/svn/memcached/trunk/server/doc/memory_management.txt (4) Advice to bounce memcache: http://www.socialtext.net/memcached/index.cgi?managing_a_memcached_cluster I'd be happy to help write up / clean up documentation...once I know how it works. If you're swamped, I can just go read the code, but that probably won't happen soon. :) Regards, Josh
Re: facebook memcached on github (redux)
> Well, there's #3 - rebalance the existing slabs using the slab > reassignment command that nobody knows about. :) Do tell! :) On the protocol side: It'd be great to get it added to http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt and eventually to the various client libraries. On the implementation side: Does there need to be a completely stale slab available? Does it / can it shuffle items around between slabs to free one up (I presume not, since that wouldn't be O(1))? Does it / can it LRU-out a bunch of items to free up a slab (also not O(1))? Without some amount of item reshuffling / forced expiring, it's not immediately clear to me how effective this would be. With the internal reshuffling, I'd be worried about in-band performance on a heavily used instance. Or perhaps I just don't know what I'm talking about. Seriously, though, it'd be super helpful to make this more visible. Just to illustrate, my initial Googling on this topic brings up: (1) Complaints about the problem: http://www.linuxjournal.com/article/7451 (2) Your client code: http://github.com/dustin/java-memcached-client/tree/slab-reassign (3) A hint that slab rebalancing is coming: http://code.sixapart.com/svn/memcached/trunk/server/doc/memory_management.txt (4) Advice to bounce memcache: http://www.socialtext.net/memcached/index.cgi?managing_a_memcached_cluster I'd be happy to help write up / clean up documentation...once I know how it works. If you're swamped, I can just go read the code, but that probably won't happen soon. :) Regards, Josh
Re: facebook memcached on github (redux)
On Dec 19, 9:17 am, "Josh Snyder" wrote: > > First, we wanted to solve the slab-ghetto problem. That is the > > situation where one slab is forcing out hot keys for lack of memory > > while another slab holds onto stale objects. > > Just wanted to give a +1 that this is an issue for others as well. At > AdMob, this has been a pretty serious inconvenience in one of our > subsystems. We've used two workarounds to date: > > (1) Bounce memcache every so often to get a clean set of slabs. > (2) Run multiple memcache instances and segregate objects > appropriately. Half our cached data is generally bounded in total > size, critical, and expensive to generate; the other half is generally > unbounded (with slow growth), not quite critical, and fairly cheap to > generate. They also inconveniently tend to live in separate slabs. > Together in one instance, this can generate very painful cache churn. > > Neither of these workarounds is ideal. Bouncing caches is unpleasant > and can require manual attention, and having different memcache > instances has a small but noticeable perf impact in the critical path. > > It would be awesome to have a proper solution to this problem. Well, there's #3 - rebalance the existing slabs using the slab reassignment command that nobody knows about. :)
Re: facebook memcached on github (redux)
On Dec 19, 3:27 am, marc kwiatkowski wrote: > I re-did facebook-memcached to make it consistent with dustin's github > repository. (No more "src" subdirectory, and I redid the commit > history so that it appears to be a branch from the original 1.2.3 > release at > http://github.com/dustin/memcached/commit/437beab948aec088dac361f27c8... > ) > > It is at > > http://github.com/fbmarc/facebook-memcached Hey, this is a great start. I wrote a tool yesterday to do content- based tree comparisons so we can see where trees diverged regardless of recorded ancestry (and messages, and authors, etc...). Yesterday, I couldn't get any variation of a facebook tree to ever match, but today I get this: http://public.west.spy.net/memcached/fbmarc-rwbin.html You can see a rather large divergence, but at the very least we can see the individual diffs that diverge. > I left the original at > > http://github.com/fbmarc/facebook-memcached-old > > I hope this will make it easier to for anyone interested to compare, > merge, and cherry-pick updates. I am a complete git n00b, so if you > have suggestions on how I can repackage this repository to make life > easier, feel free to enlighten me. Well, you can do it all in a single repo. :) My memcached repo has a number of unrelated trees (including this one now). > In the next couple of weeks I'm going to review all of the updates in > dustin's master branch and pull as much as I can into ours. I'm also > cleaning up our C UDP memcache client, php extension and proxy and > will get them up on github as soon as I get everything in presentable > shape. One thing that might make this easier for you is to just fork the repo on github and use their fork queue. I've got your master as of 689aafc9d4420e0193368cd149c94ac9524e51ce and it's good. It's a minor detail at this point, but I'd like to get the authors rewritten before merging anything back. That is, I'd like to avoid this kinds of thing: Author: ps I can clean all those up for you if you'd like, I just need a map from these: 43 ttung 11 ps 9 hzhao 1 marc 1 sky 1 nneul to real names and addresses.
Re: facebook memcached on github (redux)
> First, we wanted to solve the slab-ghetto problem. That is the > situation where one slab is forcing out hot keys for lack of memory > while another slab holds onto stale objects. Just wanted to give a +1 that this is an issue for others as well. At AdMob, this has been a pretty serious inconvenience in one of our subsystems. We've used two workarounds to date: (1) Bounce memcache every so often to get a clean set of slabs. (2) Run multiple memcache instances and segregate objects appropriately. Half our cached data is generally bounded in total size, critical, and expensive to generate; the other half is generally unbounded (with slow growth), not quite critical, and fairly cheap to generate. They also inconveniently tend to live in separate slabs. Together in one instance, this can generate very painful cache churn. Neither of these workarounds is ideal. Bouncing caches is unpleasant and can require manual attention, and having different memcache instances has a small but noticeable perf impact in the critical path. It would be awesome to have a proper solution to this problem. -josh
Re: facebook memcached on github
On Mon, Dec 15, 2008 at 9:14 PM, Tony Tung wrote: > > On 12/14/08 3:40 AM, "Anatoly Vorobey" wrote: > > Tony, > > >1. As you mentioned, we don't see any benefit to runtime selection of >the storage engine; thus the indirect call is unnecessary in our >environment. > > I'm curious about this choice. Have you tried to benchmark the benefit of > using a compile-time choice of a storage engine vs. indirect calls at > runtime? Do you happen to have any statistics on hand regarding whether that > change consistently influences latency/throughput? > > > We haven't done any benchmarks. However, as I understand Toru and Trond's > design, I suspect it would not be particularly expensive. Because the key > is at a fixed location no matter the storage engine, the assoc_find call > would not need to call into the storage engine. > Thanks. I would be surprised if using direct vs indirect calls to the storage engine registered in any meaningful way at all, even if it were done inside assoc_find. -- Anatoly Vorobey, avoro...@gmail.com http://avva.livejournal.com (Russian) http://www.lovestwell.org (English)
Re: facebook memcached on github
On 12/14/08 3:40 AM, "Anatoly Vorobey" wrote: Tony, 1. As you mentioned, we don't see any benefit to runtime selection of the storage engine; thus the indirect call is unnecessary in our environment. I'm curious about this choice. Have you tried to benchmark the benefit of using a compile-time choice of a storage engine vs. indirect calls at runtime? Do you happen to have any statistics on hand regarding whether that change consistently influences latency/throughput? We haven't done any benchmarks. However, as I understand Toru and Trond's design, I suspect it would not be particularly expensive. Because the key is at a fixed location no matter the storage engine, the assoc_find call would not need to call into the storage engine. Thanks, Tony
Re: facebook memcached on github
On Sun, Dec 14, 2008 at 3:40 AM, Anatoly Vorobey wrote: > Tony, >> >> As you mentioned, we don't see any benefit to runtime selection of the >> storage engine; thus the indirect call is unnecessary in our environment. > > I'm curious about this choice. Have you tried to benchmark the benefit of > using a compile-time choice of a storage engine vs. indirect calls at > runtime? Do you happen to have any statistics on hand regarding whether that > change consistently influences latency/throughput? If it turns out in benchmarks to really honestly be painful enough to warrant compiling with direct function calls, maybe a macro could be used that maps at compile-time either directly to one storage engine or to the run-time selected function pointer. Aaron > -- > Anatoly Vorobey, avoro...@gmail.com > http://avva.livejournal.com (Russian) > http://www.lovestwell.org (English) > >
Re: facebook memcached on github
Tony, > >1. As you mentioned, we don't see any benefit to runtime selection of >the storage engine; thus the indirect call is unnecessary in our >environment. > > I'm curious about this choice. Have you tried to benchmark the benefit of using a compile-time choice of a storage engine vs. indirect calls at runtime? Do you happen to have any statistics on hand regarding whether that change consistently influences latency/throughput? -- Anatoly Vorobey, avoro...@gmail.com http://avva.livejournal.com (Russian) http://www.lovestwell.org (English)
Re: facebook memcached on github
On 12/14/08 3:21 AM, "Tony Tung" wrote: In any case, we started our work on supporting multiple storage engines before we heard of Toru and Trond's work. When we did hear about it, we examined their proposal but found that it did not fit our model for two reasons: 1. As you mentioned, we don't see any benefit to runtime selection of the storage engine; thus the indirect call is unnecessary in our environment. 2. It does not support spreading the data out to multiple blocks as we ended up doing with our flat allocator. While it is true that it could be managed by copying the data to a contiguous buffer, that's very expensive. One more thing -- I am certainly not saying that the way we implemented multiple storage engines is The Right Way. I'm just saying that this was right for us; YMMV. :) Thanks, Tony
Re: facebook memcached on github
Comments inline. Thanks, Tony On 12/13/08 3:45 PM, "dormando" wrote: - The FB tree contains a binary protocol that was proprietary up until the release. The memcached binary protocol we have was designed and written by a consortium of users, and has existing clients. I don't think it would be difficult to bring the FB tree's binary protocol to a compatible state. It would also be pretty easy to remove it altogether, as it is pretty modular. - The FB tree contains a weird mishmash of storage engines. Flat allocator? Ok. Memory pools, ok... #ifdef's for the old engine. Fair enough. The memcached community (mainly Toru and Trond) have been working towards a pluggable storage engine interface for memcached. With the intention of pulling in a lot of these forks, and allowing us to ship multiple storage engines which can be runtime pluggable. I recall this was brought up during the hackathon, and FB had serious concerns about the usage of indirect function calls for a storage engine interface. You'd prefer to #ifdef and compile it all out in order to save a few cycles from some primary calls. This goes against what the community has been working towards. (there're even handy public drafts of the interface: http://code.google.com/p/memcached/wiki/EngineInterface). Flat allocator is a storage engine. Memory pools are for finding memory hogs/leaks. In any case, we started our work on supporting multiple storage engines before we heard of Toru and Trond's work. When we did hear about it, we examined their proposal but found that it did not fit our model for two reasons: 1. As you mentioned, we don't see any benefit to runtime selection of the storage engine; thus the indirect call is unnecessary in our environment. 2. It does not support spreading the data out to multiple blocks as we ended up doing with our flat allocator. While it is true that it could be managed by copying the data to a contiguous buffer, that's very expensive. - The FB tree contains dead ends, debug code, and dangerous features. We would never release 'flush_regex' on the main release. Not on purpose, anyway. Besides flush_regex, could you be more specific? - All of the stats output code was rewritten into long, ugly 'append_to_buffer' lines. This doesn't seem necessary, and fills in the gap of what little code wasn't rewritten elsewhere... It's actually quite relevant. If someone were to add to the stats output without adjusting the size of the output buffers, we could easily overrun them. Tracking down memory corruption is such a pain and append_to_buffer guarantees that will not happen (at least not for stats).
Re: facebook memcached
Comments inline. On 12/13/08 12:45 PM, "Aaron Stone" wrote: Branch friction aside, Paul Saab's description of the interesting problems is really something worth expanding upon and learning from in context of a real-world implementation going after C10K scalability. For example, Paul notes that with thousands of open TCP connections, memcached can start using several extra gigabytes of memory for buffers. Well, that's interesting! I want to know more about that. What can we learn here? In conn_new(..), we allocate memory for receive and transmit buffers on the order of magnitude of 10s of KB. This doesn't seem like a lot, but we've gone as high as 400k simultaneous connections. It turned out that very few connections are active at any moment in time, so if we share those buffers, we can save a ton of memory. There is still the issue of memory allocated in the kernel for TCP, but we don't have much control over that. Paul notes that they switched to using UDP for GETs with application-level flow control. Only GETs? App flow control? That's good to know -- IIRC, when memcached over UDP was last being discussed there were advocates for SET over UDP and for various approaches to flow control. (For my part, I advocated for a GET variant with an offset+length parameter.) We only do GETs over UDP. Having other operations go over UDP is not an insurmountable task, we just didn't feel like we needed to optimize those operations. As for app flow control, we found that if the tail packet(s) of a memcache response is ever dropped, Linux will not retransmit for 250ms. With UDP, we can control that retransmission. Thanks, Tony
Re: facebook memcached
I didn't mean anything negative by my post. Please don't take offense. It was more about confirming the license details in my own head. On Dec 13, 2008, at 2:53 PM, Todd Berman wrote: And even if it was GPL licensed 1) They aren't distributing their version of memcached, so there is no requirement for them to be sharing anything 2) And, even if they were distributing their work, they aren't required to merge their patches anywhere, just publish them. Go look at sony's website, you will find all sorts of interesting patches to the kernel to make it work properly on their TVs and things like that (http://products.sel.sony.com/opensource/), which is less than facebook just did. --Todd On Dec 12, 4:27 pm, mike wrote: On Fri, Dec 12, 2008 at 1:46 PM, Dustin wrote: If anyone at facebook is listening, is it possible at all to add this work onto the codebase where everyone else has been working? +1 not to mention couldn't that technically be required in the GPL/whatever license memcached is under?
Re: facebook memcached on github
Hey, > Finally, I produced these branches by running git-svn and git-filter- > branch against our internal SVN repositories. The addition of the > root "src" directory and the lack of a clear branch from 1.2.3 make > comparison with existing branches unnecessarily difficult. I > apologize for the inconvenience. I will redo the git svn extraction > to correct those issues later this week. Thank you for the clarification. Though that's similar to what our own git repo was until dustin rewrote it, and what mine presently is. What we have right now are actually three trees with unrelated history. Mine - the original cleaned up git-svn import. Dustin's, the rewritten binary tree, and now FB's, which is a weird rewrite based off of 1.2.3. So right now I have all the benefits of a nice distributed RCS, but none of the benefits of being able to manage these branches, since we're all touting to have the same thing with unrelated histories. My work in the short term is to get the binary and stable trees related again, since even if the binary protocol goes out, we will have to maintain the 1.2 series for a while. > In the mean time I hope you will let us know what you think and what > else we can do to make memcached even better. We at facebook would be > happy to discuss questions and issues about our release in this group. Sounds good. I'll reiterate the things we've gone over privately already and we'll work from there. First, for clarity, a few small bits that were discernable from the last merge request were committed and credited in the stable tree. I don't think that's been in a release yet, however. Apologies on my part. As Dustin said, the facebook tree, in its current state, is basically unmergable. For the handful of us who know the codebase well enough can readily tell that this tree is actually a rewrite. There's no items.c at all, there're new files spread about, #ifdef's everywhere. Also as Dustin said the original diffs were longer than the original codebase. During our last discussion at the hackathon in october, we talked about our inability to work with the original code dump. I had directly asked for "at least a tree with history", so we could get a *hint* of insight into what the changes were, and get smaller diffs to examine. The github repository is exactly this. I even okay'ed the usage of github instead of an SVN branch. Easier for all of us to work against. The hope was that we'd be able to work together on messaging and figuring out /what parts/ of the tree were mergable. Instead I've been reading a flood from users chattering about the amazingly performant facebook fork of memcached. - while it's an implementation of memcached provided by facebook, I'm hard pressed to consider it a fork at this point. As another poster has pointed out, FB isn't obligated to do anything, even a code dump is nice. However, most people don't put up code dumps and claim them as merge requests. Even on other projects that I effectively own now (see: MogileFS) I actually bother to rewrite topic branches into clean mail-able, documented patchsets, which go up for review. This is a project where I really don't have to do that at all, and the changes are all internal performance/feature fixes for livejournal and typepad. A lot of things stand between us and mergability: - The FB tree is not compatible with either tree, as CAS was removed. Counters are also very different, but that's minor. Trond has a patch making the CAS requirement a runtime option... I haven't examined it yet, but I recall the FB complaint was not wanting to waste 8 bytes per item storing the CAS identifier. - The FB tree contains a binary protocol that was proprietary up until the release. The memcached binary protocol we have was designed and written by a consortium of users, and has existing clients. - The FB tree contains a weird mishmash of storage engines. Flat allocator? Ok. Memory pools, ok... #ifdef's for the old engine. Fair enough. The memcached community (mainly Toru and Trond) have been working towards a pluggable storage engine interface for memcached. With the intention of pulling in a lot of these forks, and allowing us to ship multiple storage engines which can be runtime pluggable. I recall this was brought up during the hackathon, and FB had serious concerns about the usage of indirect function calls for a storage engine interface. You'd prefer to #ifdef and compile it all out in order to save a few cycles from some primary calls. This goes against what the community has been working towards. (there're even handy public drafts of the interface: http://code.google.com/p/memcached/wiki/EngineInterface). - The FB tree contains dead ends, debug code, and dangerous features. We would never release 'flush_regex' on the main release. Not on purpose, anyway. - All of the stats output code was rewritten into long, ugly 'append_to_buffer' lines. This doesn't seem necessary, and fills in the gap of what little cod
Re: facebook memcached
And even if it was GPL licensed 1) They aren't distributing their version of memcached, so there is no requirement for them to be sharing anything 2) And, even if they were distributing their work, they aren't required to merge their patches anywhere, just publish them. Go look at sony's website, you will find all sorts of interesting patches to the kernel to make it work properly on their TVs and things like that (http://products.sel.sony.com/opensource/), which is less than facebook just did. --Todd On Dec 12, 4:27 pm, mike wrote: > On Fri, Dec 12, 2008 at 1:46 PM, Dustin wrote: > > If anyone at facebook is listening, is it possible at all to add > > this work onto the codebase where everyone else has been working? > > +1 > > not to mention couldn't that technically be required in the > GPL/whatever license memcached is under?
Re: facebook memcached
On Sat, Dec 13, 2008 at 12:45:58PM -0800, Aaron Stone wrote: > For example, Paul notes that with thousands of open TCP connections, > memcached can start using several extra gigabytes of memory for > buffers. Well, that's interesting! I want to know more about that. > What can we learn here? And at a much higher level, this caught my eye: Although we improved the memory efficiency with TCP, we moved to UDP for get operations to reduce network traffic and implement application-level flow control for multi-gets (gets of hundreds of keys in parallel). I'm curious about the application design that uses *hundreds* of gets in parallel. Is that per web request? I typically have just a few fetches per request, and they don't happen in parallel -- in my code there isn't one place that knows about everything that needs to be fetched at at once. -- Bill Moseley mose...@hank.org Sent from my iMutt
Re: facebook memcached
+2. Optimism is a welcome change to this discussion, though the proposed merge-athon is good to hear, as well. On Sat, Dec 13, 2008 at 12:45 PM, Aaron Stone wrote: > > Branch friction aside, Paul Saab's description of the interesting > problems is really something worth expanding upon and learning from in > context of a real-world implementation going after C10K scalability. > > For example, Paul notes that with thousands of open TCP connections, > memcached can start using several extra gigabytes of memory for > buffers. Well, that's interesting! I want to know more about that. > What can we learn here? > > Paul notes that they switched to using UDP for GETs with > application-level flow control. Only GETs? App flow control? That's > good to know -- IIRC, when memcached over UDP was last being discussed > there were advocates for SET over UDP and for various approaches to > flow control. (For my part, I advocated for a GET variant with an > offset+length parameter.) > > So while the devs work out the details and disagreements of merging > code bases, let's also keep the communication channels open about the > engineering decisions and experiences in this work. At the end of the > day, if we haven't collectively advanced the state of the art and > updated the best practices in our field, we're doing something wrong. > > Aaron > > > On Fri, Dec 12, 2008 at 1:34 PM, steve.yen wrote: > > > > fyi, Paul Saab's note on facebook's memcached improvements and git > > repo (originally pointed out to me by Dustin) > > > > http://www.facebook.com/note.php?note_id=39391378919 > > > > >
Re: facebook memcached
Branch friction aside, Paul Saab's description of the interesting problems is really something worth expanding upon and learning from in context of a real-world implementation going after C10K scalability. For example, Paul notes that with thousands of open TCP connections, memcached can start using several extra gigabytes of memory for buffers. Well, that's interesting! I want to know more about that. What can we learn here? Paul notes that they switched to using UDP for GETs with application-level flow control. Only GETs? App flow control? That's good to know -- IIRC, when memcached over UDP was last being discussed there were advocates for SET over UDP and for various approaches to flow control. (For my part, I advocated for a GET variant with an offset+length parameter.) So while the devs work out the details and disagreements of merging code bases, let's also keep the communication channels open about the engineering decisions and experiences in this work. At the end of the day, if we haven't collectively advanced the state of the art and updated the best practices in our field, we're doing something wrong. Aaron On Fri, Dec 12, 2008 at 1:34 PM, steve.yen wrote: > > fyi, Paul Saab's note on facebook's memcached improvements and git > repo (originally pointed out to me by Dustin) > > http://www.facebook.com/note.php?note_id=39391378919 > >
Re: facebook memcached
On Dec 12, 11:50 pm, Tony Tung wrote: > > 1. On May 16, 2008, I sent an email to Dustin and Dormando, having gotten > the impression that they were the de-facto maintainers of memcached, asking > how we could merge our changes back into the mainline of development. We > even offered to host a merge-athon to assist in the process. > 2. I was asked to send our tree. I did so. > 3. Upon receiving our tree, I was asked to send our changes as a series of > diffs. > 4. I broke out major bugfixes and performance improvements into separate > diffs and sent them all of them by that evening. > 5. On July 8, 2008, having not seen any of our changes merged into the > mainline, I sent a followup email asking if there were plans to merge our > changes. > 6. At the most recent hackathon, we were asked to provide a repository with > discreet changes. We all appreciate the work you've put into this, and I don't think we've ever indicated otherwise. I don't think we need to argue too much about this (especially publicly), but between your #5 and #6 there, both of us sent you an email regarding the patch you sent named ``kitchen sink'' that was twice the size of the entire memcached codebase, replaced nearly every line everywhere, and didn't actually apply on any tree I was able to find. I'm not sure about Dormando, but today is the first I've heard from you since. > I would like to point out that our src/ subdirectory is exactly the public > memcached repository. Which public memcached repository? Without any common ancestry, no automated merges can occur. If I do a subdirectory filter I *may* be able to find a tree hash that matches a version that we've got somewhere and then reconstruct history from that, but it's kind of a lot of work, and considering I couldn't get your changes to apply in May, I doubt I'd be able to do any better today. > Finally, we're willing to host a merge-athon any time the community would > like to. We have no desire to maintain a branch of the memcached code. This would be good. Is it possible for you guys to start by building a series of changesets from a community branch (my preference would be http://github.com/dustin/memcached/tree/rewritten-bin as it's the latest I know people have been doing work on) that bring forth the essence of the work you've done?
Re: facebook memcached
Dustin, Dormando: I must say I am extremely disappointed in your response to our release of our code base. Perhaps if it were someone with less understanding of our attempts to merge our changes back, I would be more empathetic. Let me try to set the record straight: 1. On May 16, 2008, I sent an email to Dustin and Dormando, having gotten the impression that they were the de-facto maintainers of memcached, asking how we could merge our changes back into the mainline of development. We even offered to host a merge-athon to assist in the process. 2. I was asked to send our tree. I did so. 3. Upon receiving our tree, I was asked to send our changes as a series of diffs. 4. I broke out major bugfixes and performance improvements into separate diffs and sent them all of them by that evening. 5. On July 8, 2008, having not seen any of our changes merged into the mainline, I sent a followup email asking if there were plans to merge our changes. 6. At the most recent hackathon, we were asked to provide a repository with discreet changes. This is exactly what we've done today. I just wanted to address one more comment. The tree the published is entirely unrelated from the trees the rest of us are working on. There's no common ancestry or even similar directory layout. As published, it sort of puts us in a position to either reimplement everyone else's work, or reimplement the facebook work. I would like to point out that our src/ subdirectory is exactly the public memcached repository. Finally, we're willing to host a merge-athon any time the community would like to. We have no desire to maintain a branch of the memcached code. Thanks, Tony On 12/12/08 6:01 PM, "dormando" wrote: On Fri, 12 Dec 2008, mike wrote: > > On Fri, Dec 12, 2008 at 1:46 PM, Dustin wrote: > >> If anyone at facebook is listening, is it possible at all to add >> this work onto the codebase where everyone else has been working? > > +1 > > not to mention couldn't that technically be required in the > GPL/whatever license memcached is under? > memcached is BSD licensed... no goodwill is technically required. it's just considered anti-goodwill to code dump and stalk off. -Dormando
Re: facebook memcached
Fri, Dec 12, 2008 at 6:01 PM, dormando wrote: > memcached is BSD licensed... no goodwill is technically required. it's just > considered anti-goodwill to code dump and stalk off. gotcha. i wasn't sure. licenses confuse me, and i was too lazy to even see what memcached was using :) agreed though. i would think facebook would be more than happy to give back though, and i think they are trying, it just isn't very useful if it isn't in line with the main memcached distro.
Re: facebook memcached
On Fri, 12 Dec 2008, mike wrote: On Fri, Dec 12, 2008 at 1:46 PM, Dustin wrote: If anyone at facebook is listening, is it possible at all to add this work onto the codebase where everyone else has been working? +1 not to mention couldn't that technically be required in the GPL/whatever license memcached is under? memcached is BSD licensed... no goodwill is technically required. it's just considered anti-goodwill to code dump and stalk off. -Dormando
Re: facebook memcached
On Fri, Dec 12, 2008 at 1:46 PM, Dustin wrote: > If anyone at facebook is listening, is it possible at all to add > this work onto the codebase where everyone else has been working? +1 not to mention couldn't that technically be required in the GPL/whatever license memcached is under?
Re: facebook memcached
+1. I'm not up for a lecture on how to post patches to a public project right now, but that isn't the way to go, nor what we've discussed in the past. -Dormando I think the results speak for themselves, but I don't know that a merge can actually occur. The tree the published is entirely unrelated from the trees the rest of us are working on. There's no common ancestry or even similar directory layout. As published, it sort of puts us in a position to either reimplement everyone else's work, or reimplement the facebook work. If anyone at facebook is listening, is it possible at all to add this work onto the codebase where everyone else has been working? We've got a lot of bug fixes and features we'd really like to not throw away here: http://github.com/dustin/memcached/tree/rewritten-bin
Re: facebook memcached
On Dec 12, 1:34 pm, "steve.yen" wrote: > fyi, Paul Saab's note on facebook's memcached improvements and git > repo (originally pointed out to me by Dustin) > > http://www.facebook.com/note.php?note_id=39391378919 I think the results speak for themselves, but I don't know that a merge can actually occur. The tree the published is entirely unrelated from the trees the rest of us are working on. There's no common ancestry or even similar directory layout. As published, it sort of puts us in a position to either reimplement everyone else's work, or reimplement the facebook work. If anyone at facebook is listening, is it possible at all to add this work onto the codebase where everyone else has been working? We've got a lot of bug fixes and features we'd really like to not throw away here: http://github.com/dustin/memcached/tree/rewritten-bin
