Hey, The protocol.txt is canonical for memcached proper. the mysql documentation are based on a fork.
The stats are pretty confusing, I apologize. Your problem is likely slab calcification. You need to look at 'stats slabs' and not strictly 'stats items'. There're a few stats counters under 'stats slabs' that will tell you how much memory has actually been allocated: STAT active_slabs 0 STAT total_malloced 0 The 'bytes' value is the total actual bytes stored, which doesn't account for overhead in the slab allocator (typically 10-20% loss). This was improved significantly by the 1.4.25 release, which the release notes (and associated PR) discuss: https://github.com/memcached/memcached/wiki/ReleaseNotes1425#new-features You'll see the evidence of this in long running instances if the size of items changes over time. IE: lots of evictions against slab class 3, which has 5 pages assigned. Meanwhile slab class 2 has 5000 pages and never evicts. 1.4.4 isn't supported, so if you continue to have trouble I highly recommend trying the new code (and start options) first. -Dormando On Wed, 9 Dec 2015, Bill Moseley wrote: > Thank you Denis for the explanation. > > On Wed, Dec 9, 2015 at 9:15 PM, Denis Samoylov <samoi...@gmail.com> wrote: > if item expired it will not be counted as evicted. > > > I'm not clear on how items with expires time are removed. After their > expires time has past are they just considered as available space in that > slab? > > > I'm confused about "evicted_nonzero" as I see two descriptions: > > "Number of times an item which had an explicit expire time set had to be > evicted from the LRU before it expired." > https://github.com/memcached/memcached/blob/master/doc/protocol.txt#L704 > > Or > > "The time of the last evicted non-zero entry" > https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-stats-items.html > > > > > So you need to look into "stats items" > (https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-stats-items.html) > and look for evictions. If you have evictions this means that you do not > have enough memory (or memory was allocated in other slabs: memcached does > not free slabs by default, so if you for example allocated all memory for 4K > slab and do not use it anymore, e.g. everything is expired in 4K slab - > you wont be able to allocate memory for 1MB slab) > > > Well, I have lots of evictions (see below), but the server has been up for a > long time. > > > But, doesn't this mean I have almost a GB of unallocated memory that can be > used to allocate new pages for slabs? > > STAT limit_maxbytes 8,589,934,592 > STAT bytes 7,438,136,137 > > > > If that's so, why would anything be evicted? Wouldn't Memcached allocate > that memory as pages to full slab classes first? > > And once a page is allocated to a slab it is never released from that slab. > > Here's the non-zero values from "stats items". The "evicted_nonzero" shows > I'm evicting a lot of keys before they expired -- although I don't know the > rate. Seems like a cache flush would be good. > > STAT items:1:number 3778 > STAT items:1:age 11829556 > STAT items:2:number 47446517 > STAT items:2:age 11734712 > STAT items:2:evicted 3650 > STAT items:2:evicted_time 300048 > STAT items:3:number 2136100 > STAT items:3:age 11318274 > STAT items:3:evicted 1111 > STAT items:3:evicted_time 723804 > STAT items:4:number 633476 > STAT items:4:age 12003825 > STAT items:4:evicted 54471719 > STAT items:4:evicted_nonzero 54471719 > STAT items:4:evicted_time 28454 > STAT items:5:number 1267004 > STAT items:5:age 12001656 > STAT items:5:evicted 644320 > STAT items:5:evicted_nonzero 644319 > STAT items:5:evicted_time 30609 > STAT items:6:number 24140 > STAT items:6:age 10669748 > STAT items:6:evicted 2962192 > STAT items:6:evicted_nonzero 2949912 > STAT items:6:evicted_time 1362778 > STAT items:7:number 5459 > STAT items:7:age 12029692 > STAT items:7:evicted 6057558 > STAT items:7:evicted_nonzero 6057558 > STAT items:7:evicted_time 2589 > STAT items:8:number 342866 > STAT items:8:age 11184370 > STAT items:8:evicted 45767 > STAT items:8:evicted_time 847838 > STAT items:9:number 1746 > STAT items:9:age 12029668 > STAT items:9:evicted 2026223 > STAT items:9:evicted_nonzero 2026223 > STAT items:9:evicted_time 2612 > STAT items:10:number 5576 > STAT items:10:age 12029153 > STAT items:10:evicted 4268899 > STAT items:10:evicted_nonzero 4268899 > STAT items:10:evicted_time 3126 > STAT items:11:number 3328 > STAT items:11:age 12030195 > STAT items:11:evicted 4107140 > STAT items:11:evicted_nonzero 4107140 > STAT items:11:evicted_time 2084 > STAT items:12:number 11403 > STAT items:12:age 12020938 > STAT items:12:evicted 28072 > STAT items:12:evicted_nonzero 28071 > STAT items:12:evicted_time 10792 > STAT items:13:number 213106 > STAT items:13:age 12021699 > STAT items:13:evicted 12499603 > STAT items:13:evicted_nonzero 12499603 > STAT items:13:evicted_time 10586 > STAT items:14:number 8504 > STAT items:14:age 12023367 > STAT items:15:number 128175 > STAT items:15:age 12026614 > STAT items:15:evicted 76488055 > STAT items:15:evicted_nonzero 76488040 > STAT items:15:evicted_time 5689 > STAT items:16:number 9385 > STAT items:16:age 12030698 > STAT items:16:evicted 39021091 > STAT items:16:evicted_nonzero 39021079 > STAT items:16:evicted_time 1581 > STAT items:17:number 4319 > STAT items:17:age 12030087 > STAT items:17:evicted 13452454 > STAT items:17:evicted_nonzero 13452438 > STAT items:17:evicted_time 2193 > STAT items:18:number 1374 > STAT items:18:age 12029789 > STAT items:18:evicted 5698741 > STAT items:18:evicted_nonzero 5698734 > STAT items:18:evicted_time 2475 > STAT items:19:number 1103 > STAT items:19:age 12028727 > STAT items:19:evicted 2628417 > STAT items:19:evicted_nonzero 2628410 > STAT items:19:evicted_time 3548 > STAT items:20:number 3377 > STAT items:20:age 12029580 > STAT items:20:evicted 6719357 > STAT items:20:evicted_nonzero 6719317 > STAT items:20:evicted_time 2690 > STAT items:21:number 3301 > STAT items:21:age 12029410 > STAT items:21:evicted 7805240 > STAT items:21:evicted_nonzero 7805216 > STAT items:21:evicted_time 2867 > STAT items:22:number 4886 > STAT items:22:age 12029511 > STAT items:22:evicted 7876885 > STAT items:22:evicted_nonzero 7876881 > STAT items:22:evicted_time 2765 > STAT items:23:number 2999 > STAT items:23:age 12030278 > STAT items:23:evicted 6451060 > STAT items:23:evicted_nonzero 6451053 > STAT items:23:evicted_time 2002 > STAT items:24:number 1257 > STAT items:24:age 12031221 > STAT items:24:evicted 5636498 > STAT items:24:evicted_nonzero 5636493 > STAT items:24:evicted_time 1058 > STAT items:25:number 767 > STAT items:25:age 12031501 > STAT items:25:evicted 4295219 > STAT items:25:evicted_nonzero 4295216 > STAT items:25:evicted_time 783 > STAT items:26:number 607 > STAT items:26:age 12031389 > STAT items:26:evicted 2926550 > STAT items:26:evicted_nonzero 2926548 > STAT items:26:evicted_time 890 > STAT items:27:number 538 > STAT items:27:age 12031340 > STAT items:27:evicted 1855678 > STAT items:27:evicted_nonzero 1855676 > STAT items:27:evicted_time 938 > STAT items:28:number 405 > STAT items:28:age 12031000 > STAT items:28:evicted 1323273 > STAT items:28:evicted_nonzero 1323266 > STAT items:28:evicted_time 1277 > STAT items:29:number 417 > STAT items:29:age 12029537 > STAT items:29:evicted 603418 > STAT items:29:evicted_nonzero 603385 > STAT items:29:evicted_time 2741 > STAT items:30:number 222 > STAT items:30:age 12029937 > STAT items:30:evicted 466043 > STAT items:30:evicted_nonzero 466043 > STAT items:30:evicted_time 2302 > STAT items:31:number 205 > STAT items:31:age 12027283 > STAT items:31:evicted 158082 > STAT items:31:evicted_nonzero 158082 > STAT items:31:evicted_time 3535 > STAT items:32:number 150 > STAT items:32:age 12028665 > STAT items:32:evicted 93647 > STAT items:32:evicted_nonzero 93647 > STAT items:32:evicted_time 3509 > STAT items:33:number 86 > STAT items:33:age 12029029 > STAT items:33:evicted 152651 > STAT items:33:evicted_nonzero 152651 > STAT items:33:evicted_time 3222 > STAT items:34:number 95 > STAT items:34:age 12028676 > STAT items:34:evicted 37551 > STAT items:34:evicted_nonzero 37551 > STAT items:34:evicted_time 3471 > STAT items:35:number 47 > STAT items:35:age 12028704 > STAT items:35:evicted 7965 > STAT items:35:evicted_nonzero 7965 > STAT items:35:evicted_time 3599 > STAT items:36:number 266 > STAT items:36:age 12028693 > STAT items:36:evicted 20217 > STAT items:36:evicted_nonzero 20217 > STAT items:36:evicted_time 3598 > STAT items:37:number 10 > STAT items:37:age 12028607 > STAT items:37:evicted 11913 > STAT items:37:evicted_nonzero 11913 > STAT items:37:evicted_time 3564 > STAT items:38:number 6 > STAT items:38:age 12029231 > STAT items:38:evicted 3178 > STAT items:38:evicted_nonzero 3178 > STAT items:38:evicted_time 3571 > STAT items:39:number 20 > STAT items:39:age 12029280 > STAT items:40:number 78 > STAT items:40:age 12028799 > STAT items:40:evicted 69 > STAT items:40:evicted_nonzero 69 > STAT items:40:evicted_time 3446 > STAT items:41:number 25 > STAT items:41:age 12029015 > STAT items:41:evicted 5911 > STAT items:41:evicted_nonzero 5911 > STAT items:41:evicted_time 3592 > STAT items:42:number 42 > STAT items:42:age 12028792 > STAT items:42:evicted 8039 > STAT items:42:evicted_nonzero 8039 > STAT items:42:evicted_time 3475 > > -- > Bill Moseley > mose...@hank.org > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > >
