Consider cacheismo. thanks! rohitk
On Sun, Feb 26, 2012 at 2:08 PM, <memcac...@googlecode.com> wrote: > Status: New > Owner: ---- > Labels: Type-Defect Priority-Medium > > New issue 256 by psla...@wp.pl: PLEASE do not remove cachedump, better > dumping feature needed > http://code.google.com/p/**memcached/issues/detail?id=256<http://code.google.com/p/memcached/issues/detail?id=256> > > Hi Guys. > > Sorry for the long post, i wrote it to describe some very big problem with > memcached (and solution). > > My company implemented adserver that handles tens of millions impressions > daily by extensively using memcached. We use memcached both to cache data > but also for staging SQL writes. And to my knowledge it is (as of today) > only available tool that can scale writes to SQL (redis because of their > reclaim policy is totally ususable and other K-V storage tools are out of > the equation because they write data to disk). > > So we run tens of thousands of writes per minute through memcached, then > we analyze the data every minute and write/update 100-200 sql rows with > aggregated data. We scaled the server from about 40-50 requests / seconds > to more than 800 so it works great. > > But we got a problem related to LRU/"lazy" reclaim. The cache is filling > all available memory and then there are some evictions becuase they keys > have different expire time (some of them just 5 seconds, the others 24 > hours). > > As a workaround we used cachedump, to get a list of keys, then issue a GET > command so the key is immediately reclaimed. And it works, the only problem > is that we can't eg. dump whole 10 million keys, because the dump is > limited. > > To see how bad it is without this kind of "fast" reclaim - after 20-30 > hours we have about 2GB of outdated keys that occupy just one SLAB. So we > can't accumulate for different traffic patterns because all slabs are > taken. While the non-expired set is like 30mb, so 1970mb out of 2000 is a > waste. So with RAM 66 TIMES bigger than actually "needed", without > cachedump we'd still got evictions. > > So can you please make "improved dump" a much needed feature request? I > saw posts by many other people asking about this. Maybe include command > line option to turn this ON if you're concerned about security? > > If that's not appropriate place to make feature requests, can you please > direct me there. > > Maybe it will be possible to make separated low priority thread that'll > scan the key list and issue get from time to time. I'm a C++ coder, how > hard will it be to make? Would it require partiall or full lock of some > important shared resource so it'd be problematic (like whole item list). > Maybe it'd be possible to fork the process (copy on write) so it'd have > access to the whole list and then just issue GETs to parent using text > protocol? > > Thanks, > Slawomir. > >
