Re: Evictions, OOM and other troubleshooting suggestion request
Hey, Just store more data and that ratio will rise. I don't know why that stat is named "memory efficiency". You have a lot of RAM free. On Mon, 13 Feb 2023, Артём Яшков wrote: > Hello again, Dormando! > Haven't heard you for a while :) > > I have started updating dashboard for our brand-new memcached server, so I've > checked some stats so far. They look ok, 91% hits, so we will increase > amount of routes to be cached and the RAM amount also. > > On the other hand, I still don't get if it is an issue that > memcached_current_bytes / memcached_malloced_bytes * 100% has such a small > ratio (~2% of > memory is being used efficiently). > > As we have a lot more routes\pages to include to caching by memcached, I > would like to know, if these metrics somehow upgradable, so we would not waste > RAM. > > STAT limit_maxbytes ~32gb > htop shows memory is used ~ 11gb > so it less then 1 gb of real bytes used... > > Looking forward to hearing from you. > > Best wishes, > Artem Iashkov > > P.S. > I use https://github.com/prometheus/memcached_exporter to get memcached stats > to prometheus\grafanaпонедельник, 31 октября 2022 г. в 09:03:53 UTC+2, > Артём Яшков: > Hello, Dormando! > > Happy to show you actual stats after being a while on the new and > updated memcached server we've done. It is many times better, than it was > before. > The only thing is, there are thousands of errors 'store_too_large' by > now, around 8 times. Should I increase -I even more (it's 2m now)? > > stats > STAT pid 50132 > STAT uptime 2673148 > STAT time 1667199498 > STAT version 1.6.17 > STAT libevent 2.1.8-stable > STAT pointer_size 64 > STAT rusage_user 56027.845797 > STAT rusage_system 202864.587583 > STAT max_connections 4000 > STAT curr_connections 39 > STAT total_connections 120820 > STAT rejected_connections 0 > STAT connection_structures 47 > STAT response_obj_oom 0 > STAT response_obj_count 1 > STAT response_obj_bytes 65536 > STAT read_buf_count 26 > STAT read_buf_bytes 425984 > STAT read_buf_bytes_free 344064 > STAT read_buf_oom 0 > STAT reserved_fds 20 > STAT cmd_get 657453008 > STAT cmd_set 60148720 > STAT cmd_flush 719 > STAT cmd_touch 0 > STAT cmd_meta 0 > STAT get_hits 582449437 > STAT get_misses 75003571 > STAT get_expired 2587689 > STAT get_flushed 9532 > STAT delete_misses 0 > STAT delete_hits 0 > STAT incr_misses 0 > STAT incr_hits 0 > STAT decr_misses 0 > STAT decr_hits 0 > STAT cas_misses 0 > STAT cas_hits 0 > STAT cas_badval 0 > STAT touch_hits 0 > STAT touch_misses 0 > STAT store_too_large 75998 > STAT store_no_memory 0 > STAT auth_cmds 0 > STAT auth_errors 0 > STAT bytes_read 6035480065990 > STAT bytes_written 11467585205846 > STAT limit_maxbytes 32212254720 > STAT accepting_conns 1 > STAT listen_disabled_num 0 > STAT time_in_listen_disabled_us 0 > STAT threads 4 > STAT conn_yields 11876 > STAT hash_power_level 16 > STAT hash_bytes 524288 > STAT hash_is_expanding 0 > STAT slab_reassign_rescues 1607820 > STAT slab_reassign_chunk_rescues 1529479 > STAT slab_reassign_evictions_nomem 0 > STAT slab_reassign_inline_reclaim 36326 > STAT slab_reassign_busy_items 2711 > STAT slab_reassign_busy_deletes 0 > STAT slab_reassign_running 0 > STAT slabs_moved 1171688 > STAT lru_crawler_running 0 > STAT lru_crawler_starts 1978777 > STAT lru_maintainer_juggles 425476885 > STAT malloc_fails 0 > STAT log_worker_dropped 0 > STAT log_worker_written 0 > STAT log_watcher_skipped 0 > STAT log_watcher_sent 0 > STAT log_watchers 0 > STAT unexpected_napi_ids 0 > STAT round_robin_fallback 0 > STAT bytes 1687793908 > STAT curr_items 21466 > STAT total_items 61756540 > STAT slab_global_page_pool 8411 > STAT expired_unfetched 34136549 > STAT evicted_unfetched 0 > STAT evicted_active 0 > STAT evictions 0 > STAT reclaimed 29198074 > STAT crawler_reclaimed 27677584 > STAT crawler_items_checked 3573821082 > STAT lrutail_reflocked 12569 > STAT moves_to_cold 65950430 > STAT moves_to_warm 34051328 > STAT moves_within_lru 35745382 > STAT direct_reclaims 0 > STAT lru_bumps_dropped 0 > END > > четверг, 22 сентября 2022 г. в 02:28:30 UTC+3, Dormando: > > > On Tue, 20 Sep 2022, Артём Яшков wrote: > > > "Memcache returned an error: SERVER_ERROR object too large for cache" > > but when I change -I to 2m the cache stops erroring and cacheRes is > Buffer(+-355617 bytes) of size > > That's the right thing to do for that error. You should be g
Re: Evictions, OOM and other troubleshooting suggestion request
On Tue, 20 Sep 2022, Артём Яшков wrote: > "Memcache returned an error: SERVER_ERROR object too large for cache" > but when I change -I to 2m the cache stops erroring and cacheRes is > Buffer(+-355617 bytes) of size That's the right thing to do for that error. You should be good to go. Let us know how it goes with 1.6.17. > I write stats items below, haven't ever checked them before, but all stats > were stats resetted at last friday > > STAT items:1:number 257 > STAT items:1:number_hot 18 > STAT items:1:number_warm 103 > STAT items:1:number_cold 136 > STAT items:1:age_hot 515 > STAT items:1:age_warm 97 > STAT items:1:age 2791 > STAT items:1:evicted 0 > STAT items:1:evicted_nonzero 0 > STAT items:1:evicted_time 0 > STAT items:1:outofmemory 0 > STAT items:1:tailrepairs 0 > STAT items:1:reclaimed 9181 > STAT items:1:expired_unfetched 2402 > STAT items:1:evicted_unfetched 0 > STAT items:1:evicted_active 0 > STAT items:1:crawler_reclaimed 2647 > STAT items:1:crawler_items_checked 196947 > STAT items:1:lrutail_reflocked 9 > STAT items:1:moves_to_cold 75564 > STAT items:1:moves_to_warm 73936 > STAT items:1:moves_within_lru 18 > STAT items:1:direct_reclaims 0 > STAT items:1:hits_to_hot 111653 > STAT items:1:hits_to_warm 1998737 > STAT items:1:hits_to_cold 73596 > STAT items:1:hits_to_temp 0 > STAT items:2:number 1316 > STAT items:2:number_hot 264 > STAT items:2:number_warm 524 > STAT items:2:number_cold 528 > STAT items:2:age_hot 142 > STAT items:2:age_warm 336 > STAT items:2:age 2663 > STAT items:2:evicted 0 > STAT items:2:evicted_nonzero 0 > STAT items:2:evicted_time 0 > STAT items:2:outofmemory 0 > STAT items:2:tailrepairs 0 > STAT items:2:reclaimed 68910 > STAT items:2:expired_unfetched 189699 > STAT items:2:evicted_unfetched 0 > STAT items:2:evicted_active 0 > STAT items:2:crawler_reclaimed 850918 > STAT items:2:crawler_items_checked 65694884 > STAT items:2:lrutail_reflocked 27 > STAT items:2:moves_to_cold 1298481 > STAT items:2:moves_to_warm 1223960 > STAT items:2:moves_within_lru 1213898 > STAT items:2:direct_reclaims 0 > STAT items:2:hits_to_hot 2825325 > STAT items:2:hits_to_warm 9281983 > STAT items:2:hits_to_cold 1094466 > STAT items:2:hits_to_temp 0 > STAT items:3:number 1668 > STAT items:3:number_hot 281 > STAT items:3:number_warm 689 > STAT items:3:number_cold 698 > STAT items:3:age_hot 120 > STAT items:3:age_warm 450 > STAT items:3:age 598 > STAT items:3:evicted 0 > STAT items:3:evicted_nonzero 0 > STAT items:3:evicted_time 0 > STAT items:3:outofmemory 0 > STAT items:3:tailrepairs 0 > STAT items:3:reclaimed 408013 > STAT items:3:expired_unfetched 250245 > STAT items:3:evicted_unfetched 0 > STAT items:3:evicted_active 0 > STAT items:3:crawler_reclaimed 781404 > STAT items:3:crawler_items_checked 81632845 > STAT items:3:lrutail_reflocked 30 > STAT items:3:moves_to_cold 1340049 > STAT items:3:moves_to_warm 1188848 > STAT items:3:moves_within_lru 1027625 > STAT items:3:direct_reclaims 0 > STAT items:3:hits_to_hot 2979474 > STAT items:3:hits_to_warm 11040546 > STAT items:3:hits_to_cold 1169857 > STAT items:3:hits_to_temp 0 > STAT items:4:number 565 > STAT items:4:number_hot 99 > STAT items:4:number_warm 232 > STAT items:4:number_cold 234 > STAT items:4:age_hot 114 > STAT items:4:age_warm 591 > STAT items:4:age 596 > STAT items:4:evicted 0 > STAT items:4:evicted_nonzero 0 > STAT items:4:evicted_time 0 > STAT items:4:outofmemory 0 > STAT items:4:tailrepairs 0 > STAT items:4:reclaimed 145435 > STAT items:4:expired_unfetched 79928 > STAT items:4:evicted_unfetched 0 > STAT items:4:evicted_active 0 > STAT items:4:crawler_reclaimed 262589 > STAT items:4:crawler_items_checked 27867101 > STAT items:4:lrutail_reflocked 12 > STAT items:4:moves_to_cold 295224 > STAT items:4:moves_to_warm 296056 > STAT items:4:moves_within_lru 157801 > STAT items:4:direct_reclaims 0 > STAT items:4:hits_to_hot 1194336 > STAT items:4:hits_to_warm 4190006 > STAT items:4:hits_to_cold 262074 > STAT items:4:hits_to_temp 0 > STAT items:5:number 259 > STAT items:5:number_hot 52 > STAT items:5:number_warm 89 > STAT items:5:number_cold 118 > STAT items:5:age_hot 104 > STAT items:5:age_warm 347 > STAT items:5:age 598 > STAT items:5:evicted 0 > STAT items:5:evicted_nonzero 0 > STAT items:5:evicted_time 0 > STAT items:5:outofmemory 0 > STAT items:5:tailrepairs 0 > STAT items:5:reclaimed 65832 > STAT items:5:expired_unfetched 38319 > STAT items:5:evicted_unfetched 0 > STAT items:5:evicted_active 0 > STAT items:5:crawler_reclaimed 156328 > STAT items:5:crawler_items_checked 15562571 > STAT items:5:lrutail_reflocked 32 > STAT items:5:moves_to_cold 521727 > STAT items:5:moves_to_warm 461376 > STAT items:5:moves_within_lru 274768 > STAT items:5:direct_reclaims 0 > STAT items:5:hits_to_hot 543398 > STAT items:5:hits_to_warm 3123443 > STAT items:5:hits_to_cold 491230 > STAT items:5:hits_to_temp 0 > STAT items:6:number 227 > STAT items:6:number_hot 113 > STAT items:6:number_warm 54 > STAT items:6:number_cold 60 > STAT items:6:age_hot 313 > ST
Re: Evictions, OOM and other troubleshooting suggestion request
"Memcache returned an error: SERVER_ERROR object too large for cache" but when I change -I to 2m the cache stops erroring and cacheRes is Buffer(+-355617 bytes) of size I write stats items below, haven't ever checked them before, but all stats were stats resetted at last friday STAT items:1:number 257 STAT items:1:number_hot 18 STAT items:1:number_warm 103 STAT items:1:number_cold 136 STAT items:1:age_hot 515 STAT items:1:age_warm 97 STAT items:1:age 2791 STAT items:1:evicted 0 STAT items:1:evicted_nonzero 0 STAT items:1:evicted_time 0 STAT items:1:outofmemory 0 STAT items:1:tailrepairs 0 STAT items:1:reclaimed 9181 STAT items:1:expired_unfetched 2402 STAT items:1:evicted_unfetched 0 STAT items:1:evicted_active 0 STAT items:1:crawler_reclaimed 2647 STAT items:1:crawler_items_checked 196947 STAT items:1:lrutail_reflocked 9 STAT items:1:moves_to_cold 75564 STAT items:1:moves_to_warm 73936 STAT items:1:moves_within_lru 18 STAT items:1:direct_reclaims 0 STAT items:1:hits_to_hot 111653 STAT items:1:hits_to_warm 1998737 STAT items:1:hits_to_cold 73596 STAT items:1:hits_to_temp 0 STAT items:2:number 1316 STAT items:2:number_hot 264 STAT items:2:number_warm 524 STAT items:2:number_cold 528 STAT items:2:age_hot 142 STAT items:2:age_warm 336 STAT items:2:age 2663 STAT items:2:evicted 0 STAT items:2:evicted_nonzero 0 STAT items:2:evicted_time 0 STAT items:2:outofmemory 0 STAT items:2:tailrepairs 0 STAT items:2:reclaimed 68910 STAT items:2:expired_unfetched 189699 STAT items:2:evicted_unfetched 0 STAT items:2:evicted_active 0 STAT items:2:crawler_reclaimed 850918 STAT items:2:crawler_items_checked 65694884 STAT items:2:lrutail_reflocked 27 STAT items:2:moves_to_cold 1298481 STAT items:2:moves_to_warm 1223960 STAT items:2:moves_within_lru 1213898 STAT items:2:direct_reclaims 0 STAT items:2:hits_to_hot 2825325 STAT items:2:hits_to_warm 9281983 STAT items:2:hits_to_cold 1094466 STAT items:2:hits_to_temp 0 STAT items:3:number 1668 STAT items:3:number_hot 281 STAT items:3:number_warm 689 STAT items:3:number_cold 698 STAT items:3:age_hot 120 STAT items:3:age_warm 450 STAT items:3:age 598 STAT items:3:evicted 0 STAT items:3:evicted_nonzero 0 STAT items:3:evicted_time 0 STAT items:3:outofmemory 0 STAT items:3:tailrepairs 0 STAT items:3:reclaimed 408013 STAT items:3:expired_unfetched 250245 STAT items:3:evicted_unfetched 0 STAT items:3:evicted_active 0 STAT items:3:crawler_reclaimed 781404 STAT items:3:crawler_items_checked 81632845 STAT items:3:lrutail_reflocked 30 STAT items:3:moves_to_cold 1340049 STAT items:3:moves_to_warm 1188848 STAT items:3:moves_within_lru 1027625 STAT items:3:direct_reclaims 0 STAT items:3:hits_to_hot 2979474 STAT items:3:hits_to_warm 11040546 STAT items:3:hits_to_cold 1169857 STAT items:3:hits_to_temp 0 STAT items:4:number 565 STAT items:4:number_hot 99 STAT items:4:number_warm 232 STAT items:4:number_cold 234 STAT items:4:age_hot 114 STAT items:4:age_warm 591 STAT items:4:age 596 STAT items:4:evicted 0 STAT items:4:evicted_nonzero 0 STAT items:4:evicted_time 0 STAT items:4:outofmemory 0 STAT items:4:tailrepairs 0 STAT items:4:reclaimed 145435 STAT items:4:expired_unfetched 79928 STAT items:4:evicted_unfetched 0 STAT items:4:evicted_active 0 STAT items:4:crawler_reclaimed 262589 STAT items:4:crawler_items_checked 27867101 STAT items:4:lrutail_reflocked 12 STAT items:4:moves_to_cold 295224 STAT items:4:moves_to_warm 296056 STAT items:4:moves_within_lru 157801 STAT items:4:direct_reclaims 0 STAT items:4:hits_to_hot 1194336 STAT items:4:hits_to_warm 4190006 STAT items:4:hits_to_cold 262074 STAT items:4:hits_to_temp 0 STAT items:5:number 259 STAT items:5:number_hot 52 STAT items:5:number_warm 89 STAT items:5:number_cold 118 STAT items:5:age_hot 104 STAT items:5:age_warm 347 STAT items:5:age 598 STAT items:5:evicted 0 STAT items:5:evicted_nonzero 0 STAT items:5:evicted_time 0 STAT items:5:outofmemory 0 STAT items:5:tailrepairs 0 STAT items:5:reclaimed 65832 STAT items:5:expired_unfetched 38319 STAT items:5:evicted_unfetched 0 STAT items:5:evicted_active 0 STAT items:5:crawler_reclaimed 156328 STAT items:5:crawler_items_checked 15562571 STAT items:5:lrutail_reflocked 32 STAT items:5:moves_to_cold 521727 STAT items:5:moves_to_warm 461376 STAT items:5:moves_within_lru 274768 STAT items:5:direct_reclaims 0 STAT items:5:hits_to_hot 543398 STAT items:5:hits_to_warm 3123443 STAT items:5:hits_to_cold 491230 STAT items:5:hits_to_temp 0 STAT items:6:number 227 STAT items:6:number_hot 113 STAT items:6:number_warm 54 STAT items:6:number_cold 60 STAT items:6:age_hot 313 STAT items:6:age_warm 613 STAT items:6:age 1566 STAT items:6:evicted 0 STAT items:6:evicted_nonzero 0 STAT items:6:evicted_time 0 STAT items:6:outofmemory 0 STAT items:6:tailrepairs 0 STAT items:6:reclaimed 11152 STAT items:6:expired_unfetched 38647 STAT items:6:evicted_unfetched 0 STAT items:6:evicted_active 0 STAT items:6:crawler_reclaimed 141106 STAT items:6:crawler_items_checked 7355947 STAT items:6:lrutail_reflocked 11 STAT items:6:moves_t
Re: Evictions, OOM and other troubleshooting suggestion request
> Hello, Dormando, > > I'm glad you've answered! > > The goal is simple - it is to make memcached work properly as the server > cashing service in our project, as it seems to me that it's not working so by > now. What are you specifically seeing that you disagree with? I don't really want to comb 1,000 lines of stats output :) Though I think you have stats slabs but not stats items output. > My decision now is to create a new virtual server having much larger RAM > available and adjusting configs that way: > -u memcached -p 11211 -m (*max available*) -c 4000 -R 100 (as I see > conn_yields errors in stats) -I 2m (as I get errors when it is 1m or lower) What errors are you getting specifically? "item too large"? conn_yields isn't an error; if you raise -R you might cause excess latency. more RAM probably can't hurt. > Would you suggest increasing -n or lowering -f in addiction to this? Why? > Also, I am looking forward to reestimate TTL to give actual htmls, it seems > low to me now. No, probably not. If your items are large the overhead is small enough that it's not worth changing settings (and you will have to re-tune if your workload changes over time). The upgrade to 1.6.17 should fix any issues with large items. > вторник, 20 сентября 2022 г. в 11:27:02 UTC+5, Dormando: > Hey, > > I'm not sure if you have a specific question or not, since it sounds > like > you're just putting up stats and asking about what you can do better? Do > you have a goal in mind? > > Since your objects are fairly large (350-450kb) you said, and you seem > to > be seeing OOM errors, can you try upgrading to 1.6.17? A fix just went > in > fixing OOM's and excess evictions for caches with mostly large objects. > > -Dormando > > On Mon, 19 Sep 2022, Артём Яшков wrote: > > > Hello there, I am new to using memcached, but I would like to improve > the performance of out project by adjusting some of settings and > improving stats of > > out memcached usage. > > Could you please help me understand what should I change (if > anything) ? > > > > It's used mostly for containing compressed html-pages of a huge > web-app (350-450kb each) and smaller data. TTL is set to 10minutes. > > > > STAT pid 1213 > > STAT uptime 18491072 > > STAT time 1663584971 > > STAT version 1.5.16 > > STAT libevent 2.0.21-stable > > STAT pointer_size 64 > > STAT rusage_user 168879.888699 > > STAT rusage_system 628808.673825 > > STAT max_connections 4000 > > STAT curr_connections 60 > > STAT total_connections 33757 > > STAT rejected_connections 0 > > STAT connection_structures 175 > > STAT reserved_fds 20 > > STAT cmd_get 81749230 > > STAT cmd_set 7246273 > > STAT cmd_flush 54 > > STAT cmd_touch 0 > > STAT get_hits 73088366 > > STAT get_misses 8660864 > > STAT get_expired 282887 > > STAT get_flushed 536 > > STAT delete_misses 0 > > STAT delete_hits 0 > > STAT incr_misses 0 > > STAT incr_hits 0 > > STAT decr_misses 0 > > STAT decr_hits 0 > > STAT cas_misses 0 > > STAT cas_hits 0 > > STAT cas_badval 0 > > STAT touch_hits 0 > > STAT touch_misses 0 > > STAT auth_cmds 0 > > STAT auth_errors 0 > > STAT bytes_read 693399361640 > > STAT bytes_written 1452329058754 > > STAT limit_maxbytes 2147483648 > > STAT accepting_conns 1 > > STAT listen_disabled_num 0 > > STAT time_in_listen_disabled_us 0 > > STAT threads 4 > > STAT conn_yields 1545 > > STAT hash_power_level 16 > > STAT hash_bytes 524288 > > STAT hash_is_expanding 0 > > STAT slab_reassign_rescues 213233 > > STAT slab_reassign_chunk_rescues 157770 > > STAT slab_reassign_evictions_nomem 182616 > > STAT slab_reassign_inline_reclaim 182196 > > STAT slab_reassign_busy_items 184670 > > STAT slab_reassign_busy_deletes 0 > > STAT slab_reassign_running 0 > > STAT slabs_moved 118905 > > STAT lru_crawler_running 0 > > STAT lru_crawler_starts 2157221 > > STAT lru_maintainer_juggles 48468923 > > STAT malloc_fails 0 > > STAT log_worker_dropped 0 > > STAT log_worker_written 0 > > STAT log_watcher_skipped 0 > > STAT log_watcher_sent 0 > > STAT bytes 1974566295 > > STAT curr_items 1 > > STAT total_items 7459506 > > STAT slab_global_page_pool 0 > > STAT expired_unfetched 3257723 > > STAT evicted_unfetched 465989 > > STAT evicted_active 3 > > STAT evictions 515146 > > STAT reclaimed 2711477 > > STAT crawler_reclaimed 3485764 > > STAT crawler_items_checked 433929115 > > STAT lrutail_reflocked 2197 > > STAT moves_to_cold 7634486 > > STAT moves_to_warm 4325739 > > STAT mo
Re: Evictions, OOM and other troubleshooting suggestion request
I've forgotten to mension, I will also upgrade to 1.6.17, as you have suggested to me before. вторник, 20 сентября 2022 г. в 11:54:16 UTC+5, Артём Яшков: > Hello, Dormando, > > I'm glad you've answered! > > The goal is simple - it is to make memcached work properly as the server > cashing service in our project, as it seems to me that it's not working so > by now. > > My decision now is to create a new virtual server having much larger RAM > available and adjusting configs that way: > -u memcached -p 11211 -m (*max available*) -c 4000 -R 100 (as I see > conn_yields errors in stats) -I 2m (as I get errors when it is 1m or lower) > > Would you suggest increasing -n or lowering -f in addiction to this? Why? > Also, I am looking forward to reestimate TTL to give actual htmls, it > seems low to me now. > > вторник, 20 сентября 2022 г. в 11:27:02 UTC+5, Dormando: > >> Hey, >> >> I'm not sure if you have a specific question or not, since it sounds like >> you're just putting up stats and asking about what you can do better? Do >> you have a goal in mind? >> >> Since your objects are fairly large (350-450kb) you said, and you seem to >> be seeing OOM errors, can you try upgrading to 1.6.17? A fix just went in >> fixing OOM's and excess evictions for caches with mostly large objects. >> >> -Dormando >> >> On Mon, 19 Sep 2022, Артём Яшков wrote: >> >> > Hello there, I am new to using memcached, but I would like to improve >> the performance of out project by adjusting some of settings and improving >> stats of >> > out memcached usage. >> > Could you please help me understand what should I change (if anything) >> ? >> > >> > It's used mostly for containing compressed html-pages of a huge web-app >> (350-450kb each) and smaller data. TTL is set to 10minutes. >> > >> > STAT pid 1213 >> > STAT uptime 18491072 >> > STAT time 1663584971 >> > STAT version 1.5.16 >> > STAT libevent 2.0.21-stable >> > STAT pointer_size 64 >> > STAT rusage_user 168879.888699 >> > STAT rusage_system 628808.673825 >> > STAT max_connections 4000 >> > STAT curr_connections 60 >> > STAT total_connections 33757 >> > STAT rejected_connections 0 >> > STAT connection_structures 175 >> > STAT reserved_fds 20 >> > STAT cmd_get 81749230 >> > STAT cmd_set 7246273 >> > STAT cmd_flush 54 >> > STAT cmd_touch 0 >> > STAT get_hits 73088366 >> > STAT get_misses 8660864 >> > STAT get_expired 282887 >> > STAT get_flushed 536 >> > STAT delete_misses 0 >> > STAT delete_hits 0 >> > STAT incr_misses 0 >> > STAT incr_hits 0 >> > STAT decr_misses 0 >> > STAT decr_hits 0 >> > STAT cas_misses 0 >> > STAT cas_hits 0 >> > STAT cas_badval 0 >> > STAT touch_hits 0 >> > STAT touch_misses 0 >> > STAT auth_cmds 0 >> > STAT auth_errors 0 >> > STAT bytes_read 693399361640 >> > STAT bytes_written 1452329058754 >> > STAT limit_maxbytes 2147483648 <(214)%20748-3648> >> > STAT accepting_conns 1 >> > STAT listen_disabled_num 0 >> > STAT time_in_listen_disabled_us 0 >> > STAT threads 4 >> > STAT conn_yields 1545 >> > STAT hash_power_level 16 >> > STAT hash_bytes 524288 >> > STAT hash_is_expanding 0 >> > STAT slab_reassign_rescues 213233 >> > STAT slab_reassign_chunk_rescues 157770 >> > STAT slab_reassign_evictions_nomem 182616 >> > STAT slab_reassign_inline_reclaim 182196 >> > STAT slab_reassign_busy_items 184670 >> > STAT slab_reassign_busy_deletes 0 >> > STAT slab_reassign_running 0 >> > STAT slabs_moved 118905 >> > STAT lru_crawler_running 0 >> > STAT lru_crawler_starts 2157221 >> > STAT lru_maintainer_juggles 48468923 >> > STAT malloc_fails 0 >> > STAT log_worker_dropped 0 >> > STAT log_worker_written 0 >> > STAT log_watcher_skipped 0 >> > STAT log_watcher_sent 0 >> > STAT bytes 1974566295 >> > STAT curr_items 1 >> > STAT total_items 7459506 >> > STAT slab_global_page_pool 0 >> > STAT expired_unfetched 3257723 >> > STAT evicted_unfetched 465989 >> > STAT evicted_active 3 >> > STAT evictions 515146 >> > STAT reclaimed 2711477 >> > STAT crawler_reclaimed 3485764 >> > STAT crawler_items_checked 433929115 >> > STAT lrutail_reflocked 2197 >> > STAT moves_to_cold 7634486 >> > STAT moves_to_warm 4325739 >> > STAT moves_within_lru 4284211 >> > STAT direct_reclaims 557604 >> > STAT lru_bumps_dropped 0 >> > >> > stats settings >> > STAT maxbytes 2147483648 <(214)%20748-3648> >> > STAT maxconns 4000 >> > STAT tcpport 11211 >> > STAT udpport 0 >> > STAT inter NULL >> > STAT verbosity 0 >> > STAT oldest 18482584 >> > STAT evictions on >> > STAT domain_socket NULL >> > STAT umask 700 >> > STAT growth_factor 1.25 >> > STAT chunk_size 48 >> > STAT num_threads 4 >> > STAT num_threads_per_udp 4 >> > STAT stat_key_prefix : >> > STAT detail_enabled no >> > STAT reqs_per_event 20 >> > STAT cas_enabled yes >> > STAT tcp_backlog 1024 >> > STAT binding_protocol auto-negotiate >> > STAT auth_enabled_sasl no >> > STAT auth_enabled_ascii no >> > STAT item_size_max 13421772
Re: Evictions, OOM and other troubleshooting suggestion request
Hello, Dormando, I'm glad you've answered! The goal is simple - it is to make memcached work properly as the server cashing service in our project, as it seems to me that it's not working so by now. My decision now is to create a new virtual server having much larger RAM available and adjusting configs that way: -u memcached -p 11211 -m (*max available*) -c 4000 -R 100 (as I see conn_yields errors in stats) -I 2m (as I get errors when it is 1m or lower) Would you suggest increasing -n or lowering -f in addiction to this? Why? Also, I am looking forward to reestimate TTL to give actual htmls, it seems low to me now. вторник, 20 сентября 2022 г. в 11:27:02 UTC+5, Dormando: > Hey, > > I'm not sure if you have a specific question or not, since it sounds like > you're just putting up stats and asking about what you can do better? Do > you have a goal in mind? > > Since your objects are fairly large (350-450kb) you said, and you seem to > be seeing OOM errors, can you try upgrading to 1.6.17? A fix just went in > fixing OOM's and excess evictions for caches with mostly large objects. > > -Dormando > > On Mon, 19 Sep 2022, Артём Яшков wrote: > > > Hello there, I am new to using memcached, but I would like to improve > the performance of out project by adjusting some of settings and improving > stats of > > out memcached usage. > > Could you please help me understand what should I change (if anything) ? > > > > It's used mostly for containing compressed html-pages of a huge web-app > (350-450kb each) and smaller data. TTL is set to 10minutes. > > > > STAT pid 1213 > > STAT uptime 18491072 > > STAT time 1663584971 > > STAT version 1.5.16 > > STAT libevent 2.0.21-stable > > STAT pointer_size 64 > > STAT rusage_user 168879.888699 > > STAT rusage_system 628808.673825 > > STAT max_connections 4000 > > STAT curr_connections 60 > > STAT total_connections 33757 > > STAT rejected_connections 0 > > STAT connection_structures 175 > > STAT reserved_fds 20 > > STAT cmd_get 81749230 > > STAT cmd_set 7246273 > > STAT cmd_flush 54 > > STAT cmd_touch 0 > > STAT get_hits 73088366 > > STAT get_misses 8660864 > > STAT get_expired 282887 > > STAT get_flushed 536 > > STAT delete_misses 0 > > STAT delete_hits 0 > > STAT incr_misses 0 > > STAT incr_hits 0 > > STAT decr_misses 0 > > STAT decr_hits 0 > > STAT cas_misses 0 > > STAT cas_hits 0 > > STAT cas_badval 0 > > STAT touch_hits 0 > > STAT touch_misses 0 > > STAT auth_cmds 0 > > STAT auth_errors 0 > > STAT bytes_read 693399361640 > > STAT bytes_written 1452329058754 > > STAT limit_maxbytes 2147483648 <(214)%20748-3648> > > STAT accepting_conns 1 > > STAT listen_disabled_num 0 > > STAT time_in_listen_disabled_us 0 > > STAT threads 4 > > STAT conn_yields 1545 > > STAT hash_power_level 16 > > STAT hash_bytes 524288 > > STAT hash_is_expanding 0 > > STAT slab_reassign_rescues 213233 > > STAT slab_reassign_chunk_rescues 157770 > > STAT slab_reassign_evictions_nomem 182616 > > STAT slab_reassign_inline_reclaim 182196 > > STAT slab_reassign_busy_items 184670 > > STAT slab_reassign_busy_deletes 0 > > STAT slab_reassign_running 0 > > STAT slabs_moved 118905 > > STAT lru_crawler_running 0 > > STAT lru_crawler_starts 2157221 > > STAT lru_maintainer_juggles 48468923 > > STAT malloc_fails 0 > > STAT log_worker_dropped 0 > > STAT log_worker_written 0 > > STAT log_watcher_skipped 0 > > STAT log_watcher_sent 0 > > STAT bytes 1974566295 > > STAT curr_items 1 > > STAT total_items 7459506 > > STAT slab_global_page_pool 0 > > STAT expired_unfetched 3257723 > > STAT evicted_unfetched 465989 > > STAT evicted_active 3 > > STAT evictions 515146 > > STAT reclaimed 2711477 > > STAT crawler_reclaimed 3485764 > > STAT crawler_items_checked 433929115 > > STAT lrutail_reflocked 2197 > > STAT moves_to_cold 7634486 > > STAT moves_to_warm 4325739 > > STAT moves_within_lru 4284211 > > STAT direct_reclaims 557604 > > STAT lru_bumps_dropped 0 > > > > stats settings > > STAT maxbytes 2147483648 <(214)%20748-3648> > > STAT maxconns 4000 > > STAT tcpport 11211 > > STAT udpport 0 > > STAT inter NULL > > STAT verbosity 0 > > STAT oldest 18482584 > > STAT evictions on > > STAT domain_socket NULL > > STAT umask 700 > > STAT growth_factor 1.25 > > STAT chunk_size 48 > > STAT num_threads 4 > > STAT num_threads_per_udp 4 > > STAT stat_key_prefix : > > STAT detail_enabled no > > STAT reqs_per_event 20 > > STAT cas_enabled yes > > STAT tcp_backlog 1024 > > STAT binding_protocol auto-negotiate > > STAT auth_enabled_sasl no > > STAT auth_enabled_ascii no > > STAT item_size_max 134217728 > > STAT maxconns_fast yes > > STAT hashpower_init 0 > > STAT slab_reassign yes > > STAT slab_automove 1 > > STAT slab_automove_ratio 0.80 > > STAT slab_automove_window 30 > > STAT slab_chunk_max 524288 > > STAT lru_crawler yes > > STAT lru_crawler_sleep 100 > > STAT lru_crawler_tocrawl 0 > > STAT tail_repair_time 0 > > STAT flush_enabled yes > > STAT dump_enabled yes > > STAT hash_algorithm murmur3 > > STAT lru_maintainer_thread yes
Re: Evictions, OOM and other troubleshooting suggestion request
Hey, I'm not sure if you have a specific question or not, since it sounds like you're just putting up stats and asking about what you can do better? Do you have a goal in mind? Since your objects are fairly large (350-450kb) you said, and you seem to be seeing OOM errors, can you try upgrading to 1.6.17? A fix just went in fixing OOM's and excess evictions for caches with mostly large objects. -Dormando On Mon, 19 Sep 2022, Артём Яшков wrote: > Hello there, I am new to using memcached, but I would like to improve the > performance of out project by adjusting some of settings and improving stats > of > out memcached usage. > Could you please help me understand what should I change (if anything) ? > > It's used mostly for containing compressed html-pages of a huge web-app > (350-450kb each) and smaller data. TTL is set to 10minutes. > > STAT pid 1213 > STAT uptime 18491072 > STAT time 1663584971 > STAT version 1.5.16 > STAT libevent 2.0.21-stable > STAT pointer_size 64 > STAT rusage_user 168879.888699 > STAT rusage_system 628808.673825 > STAT max_connections 4000 > STAT curr_connections 60 > STAT total_connections 33757 > STAT rejected_connections 0 > STAT connection_structures 175 > STAT reserved_fds 20 > STAT cmd_get 81749230 > STAT cmd_set 7246273 > STAT cmd_flush 54 > STAT cmd_touch 0 > STAT get_hits 73088366 > STAT get_misses 8660864 > STAT get_expired 282887 > STAT get_flushed 536 > STAT delete_misses 0 > STAT delete_hits 0 > STAT incr_misses 0 > STAT incr_hits 0 > STAT decr_misses 0 > STAT decr_hits 0 > STAT cas_misses 0 > STAT cas_hits 0 > STAT cas_badval 0 > STAT touch_hits 0 > STAT touch_misses 0 > STAT auth_cmds 0 > STAT auth_errors 0 > STAT bytes_read 693399361640 > STAT bytes_written 1452329058754 > STAT limit_maxbytes 2147483648 > STAT accepting_conns 1 > STAT listen_disabled_num 0 > STAT time_in_listen_disabled_us 0 > STAT threads 4 > STAT conn_yields 1545 > STAT hash_power_level 16 > STAT hash_bytes 524288 > STAT hash_is_expanding 0 > STAT slab_reassign_rescues 213233 > STAT slab_reassign_chunk_rescues 157770 > STAT slab_reassign_evictions_nomem 182616 > STAT slab_reassign_inline_reclaim 182196 > STAT slab_reassign_busy_items 184670 > STAT slab_reassign_busy_deletes 0 > STAT slab_reassign_running 0 > STAT slabs_moved 118905 > STAT lru_crawler_running 0 > STAT lru_crawler_starts 2157221 > STAT lru_maintainer_juggles 48468923 > STAT malloc_fails 0 > STAT log_worker_dropped 0 > STAT log_worker_written 0 > STAT log_watcher_skipped 0 > STAT log_watcher_sent 0 > STAT bytes 1974566295 > STAT curr_items 1 > STAT total_items 7459506 > STAT slab_global_page_pool 0 > STAT expired_unfetched 3257723 > STAT evicted_unfetched 465989 > STAT evicted_active 3 > STAT evictions 515146 > STAT reclaimed 2711477 > STAT crawler_reclaimed 3485764 > STAT crawler_items_checked 433929115 > STAT lrutail_reflocked 2197 > STAT moves_to_cold 7634486 > STAT moves_to_warm 4325739 > STAT moves_within_lru 4284211 > STAT direct_reclaims 557604 > STAT lru_bumps_dropped 0 > > stats settings > STAT maxbytes 2147483648 > STAT maxconns 4000 > STAT tcpport 11211 > STAT udpport 0 > STAT inter NULL > STAT verbosity 0 > STAT oldest 18482584 > STAT evictions on > STAT domain_socket NULL > STAT umask 700 > STAT growth_factor 1.25 > STAT chunk_size 48 > STAT num_threads 4 > STAT num_threads_per_udp 4 > STAT stat_key_prefix : > STAT detail_enabled no > STAT reqs_per_event 20 > STAT cas_enabled yes > STAT tcp_backlog 1024 > STAT binding_protocol auto-negotiate > STAT auth_enabled_sasl no > STAT auth_enabled_ascii no > STAT item_size_max 134217728 > STAT maxconns_fast yes > STAT hashpower_init 0 > STAT slab_reassign yes > STAT slab_automove 1 > STAT slab_automove_ratio 0.80 > STAT slab_automove_window 30 > STAT slab_chunk_max 524288 > STAT lru_crawler yes > STAT lru_crawler_sleep 100 > STAT lru_crawler_tocrawl 0 > STAT tail_repair_time 0 > STAT flush_enabled yes > STAT dump_enabled yes > STAT hash_algorithm murmur3 > STAT lru_maintainer_thread yes > STAT lru_segmented yes > STAT hot_lru_pct 20 > STAT warm_lru_pct 40 > STAT hot_max_factor 0.20 > STAT warm_max_factor 2.00 > STAT temp_lru no > STAT temporary_ttl 61 > STAT idle_timeout 0 > STAT watcher_logbuf_size 262144 > STAT worker_logbuf_size 65536 > STAT track_sizes no > STAT inline_ascii_response no > > STAT 1:chunk_size 96 > STAT 1:chunks_per_page 10922 > STAT 1:total_pages 1 > STAT 1:total_chunks 10922 > STAT 1:used_chunks 352 > STAT 1:free_chunks 10570 > STAT 1:free_chunks_end 0 > STAT 1:mem_requested 31855 > STAT 1:get_hits 1291370 > STAT 1:cmd_set 9428 > STAT 1:delete_hits 0 > STAT 1:incr_hits 0 > STAT 1:decr_hits 0 > STAT 1:cas_hits 0 > STAT 1:cas_badval 0 > STAT 1:touch_hits 0 > STAT 2:chunk_size 120 > STAT 2:chunks_per_page 8738 > STAT 2:total_pages 1 > STAT 2:total_chunks 8738 > STAT 2:used_chunks 1573 > STAT 2:free_chunks 7165 > STAT 2:free_chunks_end 0 > STAT 2:mem_requested 182484 > STAT 2:get_hits 7988943 > STAT 2:cmd_set 612246 > STAT 2:delete_hits 0 > STAT 2:incr_h
