Re: measuring memcache efficiency
On Fri, 4 Dec 2009, aprildtd wrote: > Thanks Dormando, that was perfect. My apologies Matt, you gave me the > correct information had I been running 1.4.x, but I didn't see those > stats in 1.2.8. Makes much more sense now. > > Keep in mind guys, I'm not trying to disparage memcache, just trying > to make sure I understand and can track all of the variables at play > here. > > Last thing Dormando, would you agree that it's not possible to compute > this information without the mem_requested stat? More or less, yes. You can roughly calculate it globally by looking at the bytes stats over the limit_maxbytes.. once you've ran out of pages to allocate. That works in earlier versions. 1.4 gives you per-slab focus.
Re: measuring memcache efficiency
Thanks Dormando, that was perfect. My apologies Matt, you gave me the correct information had I been running 1.4.x, but I didn't see those stats in 1.2.8. Makes much more sense now. Keep in mind guys, I'm not trying to disparage memcache, just trying to make sure I understand and can track all of the variables at play here. Last thing Dormando, would you agree that it's not possible to compute this information without the mem_requested stat? On Dec 4, 3:32 pm, dormando wrote: > > Thanks Matt. I did look a bit through the 'stats slabs' command, but > > perhaps I'm not interpreting it correctly. In the most basic test, > > when I put in a 100 byte object, I'm seeing a slab being created with > > a chunk size of 176. However, 'stats sizes' shows me one item of > > '192'. So there's part of my confusion...am I using 176 bytes for the > > object or 192? > > > The second part of my confusion is the ability to actually see that > > 100 byte object. If instead of 100 bytes, I use 150, I'm not seeing > > any difference in the output of 'stats slabs' or 'stats sizes'. > > Obviously I can do these contrived tests and know what it is I'm > > putting into the cache, but I'm concerned that when it moves into a > > production setting I won't know the exact size of all the objects in > > the cache. I'm using server version 1.2.8 at the moment. > > > Am I reading these stats incorrectly? > > > Any detailed help would be really appreciated. > > > Thanks so much. > > An item size is the value length + the key length + pointers + bytes to > store the length + CAS header + a couple terminators/misc things. I don't > have the exact item overhead offhand but will look it up and put it in the > wiki. > > You can easily calculate your memory overhead on a slab: > > STAT 3:chunk_size 152 > STAT 3:chunks_per_page 6898 > STAT 3:total_pages 291 > STAT 3:total_chunks 2007318 > STAT 3:used_chunks 2007310 > STAT 3:free_chunks 8 > STAT 3:free_chunks_end 0 > STAT 3:mem_requested 271013713 > > chunk_size * chunks_per_page is the amount of bytes in a page for this > slab class, which is 1048496 here. > > * 291 pages == 305112336 bytes allocated in the slab. > > mem_requested is a shorthand that states the amount of memory actual items > (the total length, value + key + misc) take up within a slab. > > 271013713 / 305112336 > ~0.89% rounded. > > So I've lost 11% memory overhead in this slab on top of the ~30 bytes per > item. used_chunks * the standard overhead will give you most of the rest > of the memory overhead. So it's probably closer to 60 megabytes, total? > > 'stats sizes' will throw items into the nearest "slab" if everything were > cut by 64 byte slabs. The rounding is probably putting it into the 192 > byte bucket for you. If your item goes into the 172 byte slab, you're > definitely using 172 bytes or less. > > The idea is that we trade off some memory overhead for consistent speed in > O(1) operations. We know ways to improve the efficiency and will be doing > so over the next few months, but I wouldn't say this is horrific at all. > Remove some headers, switch back to malloc or jemalloc/etc and you lose > consistent performance. > > The overhead is most pronounced for small keys as well. Consider reducing > your key size, disabling CAS (-C) if you never use it (8 bytes per item), > or reducing the slab growth factor to close down the overhead. > > As soon as I get a chance I'm adding some more modes to damemtop so folks > can more easily see slab overhead... The mem_requested stat trond added > and the pointer_size stat lets us trivially calculate overhead given that > you already understand that a stored "value" is actually "key + value" for > length, not just the value you're storing. > > I'll throw out a side pointer here actually; This sort of knowledge is why > it's nice that memcached can store 0 byte values. If your client allows > you, you can store bits in the flags section, otherwise the existence of > the key itself may be enough data for some things you store. > > -Dormando
Re: measuring memcache efficiency
On Dec 4, 3:32 pm, dormando wrote: > An item size is the value length + the key length + pointers + bytes to > store the length + CAS header + a couple terminators/misc things. I don't > have the exact item overhead offhand but will look it up and put it in the > wiki. The builds emit this data -- it varies by platform somewhat, but the ``sizes'' command in the source tree spits this out (64-bit ubuntu): Slab Stats 40 Thread stats176 Global stats112 Settings128 Item (no cas) 16 Item (cas) 24 Libevent thread 192 Connection 520 libevent thread cumulative 8368 Thread stats cumulative 8176
Re: measuring memcache efficiency
> Thanks Matt. I did look a bit through the 'stats slabs' command, but > perhaps I'm not interpreting it correctly. In the most basic test, > when I put in a 100 byte object, I'm seeing a slab being created with > a chunk size of 176. However, 'stats sizes' shows me one item of > '192'. So there's part of my confusion...am I using 176 bytes for the > object or 192? > > The second part of my confusion is the ability to actually see that > 100 byte object. If instead of 100 bytes, I use 150, I'm not seeing > any difference in the output of 'stats slabs' or 'stats sizes'. > Obviously I can do these contrived tests and know what it is I'm > putting into the cache, but I'm concerned that when it moves into a > production setting I won't know the exact size of all the objects in > the cache. I'm using server version 1.2.8 at the moment. > > Am I reading these stats incorrectly? > > Any detailed help would be really appreciated. > > Thanks so much. An item size is the value length + the key length + pointers + bytes to store the length + CAS header + a couple terminators/misc things. I don't have the exact item overhead offhand but will look it up and put it in the wiki. You can easily calculate your memory overhead on a slab: STAT 3:chunk_size 152 STAT 3:chunks_per_page 6898 STAT 3:total_pages 291 STAT 3:total_chunks 2007318 STAT 3:used_chunks 2007310 STAT 3:free_chunks 8 STAT 3:free_chunks_end 0 STAT 3:mem_requested 271013713 chunk_size * chunks_per_page is the amount of bytes in a page for this slab class, which is 1048496 here. * 291 pages == 305112336 bytes allocated in the slab. mem_requested is a shorthand that states the amount of memory actual items (the total length, value + key + misc) take up within a slab. 271013713 / 305112336 ~0.89% rounded. So I've lost 11% memory overhead in this slab on top of the ~30 bytes per item. used_chunks * the standard overhead will give you most of the rest of the memory overhead. So it's probably closer to 60 megabytes, total? 'stats sizes' will throw items into the nearest "slab" if everything were cut by 64 byte slabs. The rounding is probably putting it into the 192 byte bucket for you. If your item goes into the 172 byte slab, you're definitely using 172 bytes or less. The idea is that we trade off some memory overhead for consistent speed in O(1) operations. We know ways to improve the efficiency and will be doing so over the next few months, but I wouldn't say this is horrific at all. Remove some headers, switch back to malloc or jemalloc/etc and you lose consistent performance. The overhead is most pronounced for small keys as well. Consider reducing your key size, disabling CAS (-C) if you never use it (8 bytes per item), or reducing the slab growth factor to close down the overhead. As soon as I get a chance I'm adding some more modes to damemtop so folks can more easily see slab overhead... The mem_requested stat trond added and the pointer_size stat lets us trivially calculate overhead given that you already understand that a stored "value" is actually "key + value" for length, not just the value you're storing. I'll throw out a side pointer here actually; This sort of knowledge is why it's nice that memcached can store 0 byte values. If your client allows you, you can store bits in the flags section, otherwise the existence of the key itself may be enough data for some things you store. -Dormando
Re: measuring memcache efficiency
Thanks Matt. I did look a bit through the 'stats slabs' command, but perhaps I'm not interpreting it correctly. In the most basic test, when I put in a 100 byte object, I'm seeing a slab being created with a chunk size of 176. However, 'stats sizes' shows me one item of '192'. So there's part of my confusion...am I using 176 bytes for the object or 192? The second part of my confusion is the ability to actually see that 100 byte object. If instead of 100 bytes, I use 150, I'm not seeing any difference in the output of 'stats slabs' or 'stats sizes'. Obviously I can do these contrived tests and know what it is I'm putting into the cache, but I'm concerned that when it moves into a production setting I won't know the exact size of all the objects in the cache. I'm using server version 1.2.8 at the moment. Am I reading these stats incorrectly? Any detailed help would be really appreciated. Thanks so much. On Dec 4, 12:25 pm, Matt Ingenthron wrote: > Hi aprildtd, > > aprildtd wrote: > > Hello all. I'm just beginning to play with memcache, and I read > > somewhere that talked about some inefficiencies with the slab > > allocator. I'm not too concerned at the moment, but I was wondering > > if you could help me monitor that? I've looked through some of the > > basic stats and can see how much memory has been allocated, but I > > can't see how much is actually being used. > > You may want to look at "stats slabs". > > http://code.google.com/p/memcached/wiki/FAQ#How_does_memcached%27s_me...http://github.com/memcached/memcached/blob/master/doc/protocol.txt#L468 > > Hope that helps, > > - Matt
Re: measuring memcache efficiency
Hi aprildtd, aprildtd wrote: Hello all. I'm just beginning to play with memcache, and I read somewhere that talked about some inefficiencies with the slab allocator. I'm not too concerned at the moment, but I was wondering if you could help me monitor that? I've looked through some of the basic stats and can see how much memory has been allocated, but I can't see how much is actually being used. You may want to look at "stats slabs". http://code.google.com/p/memcached/wiki/FAQ#How_does_memcached%27s_memory_allocation_work?_Why_not_use_ma http://github.com/memcached/memcached/blob/master/doc/protocol.txt#L468 Hope that helps, - Matt
measuring memcache efficiency
Hello all. I'm just beginning to play with memcache, and I read somewhere that talked about some inefficiencies with the slab allocator. I'm not too concerned at the moment, but I was wondering if you could help me monitor that? I've looked through some of the basic stats and can see how much memory has been allocated, but I can't see how much is actually being used. Thanks so much guys!
