Hi, I'm investigating the Linux load balancer, meanwhile, I'm trying to understand what is happening in Memcached and just noticed the different number of Memcached threads on my machines. Linux doesn't provide immediate access to generally latency-sensitive threads/processes like Memcached worker threads, causing HoL and eventually long-tail latency, not a new thing though.
But before that, I should know what worker threads I need to consider. You gave me some pointers to different internal threads of Memcached though. Even intermittent occurrence of some internal worker threads of Memcached(rebalancer, crawler or ...) might block some requests (I'm not sure if that's the case or not). ULE scheduler sounds suffering from the same flaw. The connection dispatcher is there, but I don't have so many connections creation, I have a good number of clients, and send requests at some rates until the end of the experiment. I'm using mutilate as my workload generator. Sure, I can check whether those are idle or not, I'm actually recording everything from `/proc/<memcached-pid>/..../stat` so the status is also available there. It is a true fact that internal threads are IDLE most of the time and it is some sort of visible in the plot, but I just want to make sure that all mostly idle ones are just internal threads and not workers. As you said some times workloads are not loaded evenly making the worker threads more difficult to distinguish. I think this doesn't matter now. The main issue here is that I have 6 threads on my big machine and 10 threads on my small machine, while I have the same Memcached configuration for both machines. I have attached the numbers for the two machines. Thanks, Alireza On Sun, Dec 15, 2019 at 3:40 PM dormando <dorma...@rydia.net> wrote: > What're you trying to accomplish? > > Can you include the output of "stats" and "stats settings" on both > machines? > > Dumb question but you've looked at the output of `ps auxH`? If just using > htop you may not see the threads that're idle. > > TCP connections are pinned to a specific worker thread on connection. > Trivial benchmarks may not load the worker threads evenly, as the > connections are handed to threads evenly via round robin. > > On Sun, 15 Dec 2019, Alireza Sanaee wrote: > > > Hi, > > Thank you for the information, > > > > Sorry for miss using the word there, yes that's all threads. I'm using > the Memcached 1.5.20. I build it myself and then run my > experiments($MEMCACHED -u > > root -p 11211 -m $MAXMEM -c 1024 -t $MEMCACHED_THREADS). And I'm > checking the number of Memcached threads in htop output. It showed me 10 > threads(workers > > included) in one machine and 6 threads(workers included) on the other > one. > > > > To share some more information, I have 200GB of memory for the bigger > machine that creates only 6 threads, and we have only 16GB of memory for > the machine > > that creates 10 threads. I'm just thinking maybe because the smaller > machine has less amount of space, and I'm actually filling in up to 15GB > then I might > > have more work to do and creates more threads. > > > > According to your information, I should expect at least 5 threads other > than the main workers. So 10 threads look OK, but how about the bigger > machine > > which spawns only 6 threads? > > > > I also had difficulties in detecting the worker threads that respond to > GET/SET requests on my results, I have attached two pictures, one of them > shows > > the actual location of each worker on various cores, and the second one > is showing userspace time spent for each worker. Apparently worker thread > number > > 1,2,4 and 5 have spent more time in userspace, so I'm concluding here > that 1,2,4 and 5 are my actual worker threads, and worker 3 and 6 are just > internal > > worker threads of Memcached. Does that make sense to you? > > > > Thanks, > > Alireza > > > > > > On Sun, Dec 15, 2019 at 7:19 AM dormando <dorma...@rydia.net> wrote: > > What version of memcached is on each machine? > > > > memcached doesn't use processes, it's multi-threaded. Different > versions > > may have a different number of background threads. In the latest > version > > there should be at least: > > > > - listener thread (main "process") > > - N worker threads > > - hash table maintenance thread > > - async log thread (for `watch` commands) > > - LRU maintainer thread > > - LRU crawler thread > > - slab rebalancer thread > > > > they're all idle unless they need to do work. LRU maintenance > thread is > > probably the most active, since it executes LRU maintenance work > deferred > > from the worker threads. Older versions have some of these > threads, but > > they were not enabled by default until 1.5.0. > > > > -Dormando > > > > On Sat, 14 Dec 2019, Alireza Sanaee wrote: > > > > > Hello, > > > I'm running Memcached on two different machines with different > specifications. And I specify the number of worker threads = 4 for both > > machines. However, > > > the number of child processes of the Memcached server is > different on two machines. On one of them, I have 6 Memcached child > processes, and > > on the other > > > server, I have 10 Memcached child processes. I'm curious to > understand how many children processes Memcached is basically spawning other > > than the worker > > > threads, and for what tasks? > > > > > > I expect the Memcached to spawn only 4 children processes or a > certain number of children processes on two machines, however, it seems not > > true. > > > > > > Thanks, > > > Alireza > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to memcached+unsubscr...@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/memcached/da7f492f-c12d-4763-86fc-d07311d21c5a%40googlegroups.com > . > > > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to a topic in > the Google Groups "memcached" group. > > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe. > > To unsubscribe from this group and all its topics, send an email > to memcached+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912141516160.3156%40dskull > . > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to memcached+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/memcached/CAL%3D0poZ-A0yCr%3DLAAoqcC041bEfC9rthdVZRHCVV-f7xUJ-e4g%40mail.gmail.com > . > > > > > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "memcached" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > memcached+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912142337470.3156%40dskull > . > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/CAL%3D0pobgWko2vTq9PcsARZFJ_QWm9-7S3PvbmYc863WYXcM7zg%40mail.gmail.com.
STAT pid 16163 STAT uptime 60 STAT time 1576425028 STAT version 1.5.20 STAT libevent 2.1.8-stable STAT pointer_size 64 STAT rusage_user 15.200371 STAT rusage_system 36.188149 STAT max_connections 1024 STAT curr_connections 12 STAT total_connections 13 STAT rejected_connections 0 STAT connection_structures 13 STAT reserved_fds 20 STAT cmd_get 0 STAT cmd_set 4514083 STAT cmd_flush 0 STAT cmd_touch 0 STAT cmd_meta 0 STAT get_hits 0 STAT get_misses 0 STAT get_expired 0 STAT get_flushed 0 STAT delete_misses 0 STAT delete_hits 0 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT touch_hits 0 STAT touch_misses 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 5042255667 STAT bytes_written 36112664 STAT limit_maxbytes 12582912000 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT time_in_listen_disabled_us 0 STAT threads 4 STAT conn_yields 150473 STAT hash_power_level 22 STAT hash_bytes 33554432 STAT hash_is_expanding 0 STAT slab_reassign_rescues 0 STAT slab_reassign_chunk_rescues 0 STAT slab_reassign_evictions_nomem 0 STAT slab_reassign_inline_reclaim 0 STAT slab_reassign_busy_items 0 STAT slab_reassign_busy_deletes 0 STAT slab_reassign_running 0 STAT slabs_moved 0 STAT lru_crawler_running 0 STAT lru_crawler_starts 255 STAT lru_maintainer_juggles 71157 STAT malloc_fails 0 STAT log_worker_dropped 0 STAT log_worker_written 0 STAT log_watcher_skipped 0 STAT log_watcher_sent 0 STAT bytes 5231824515 STAT curr_items 4514085 STAT total_items 4514085 STAT slab_global_page_pool 0 STAT expired_unfetched 0 STAT evicted_unfetched 0 STAT evicted_active 0 STAT evictions 0 STAT reclaimed 0 STAT crawler_reclaimed 0 STAT crawler_items_checked 0 STAT lrutail_reflocked 0 STAT moves_to_cold 3611268 STAT moves_to_warm 0 STAT moves_within_lru 0 STAT direct_reclaims 0 STAT lru_bumps_dropped 0 END
STAT pid 4888 STAT uptime 9834 STAT time 1576419223 STAT version 1.4.15 STAT libevent 2.0.21-stable STAT pointer_size 64 STAT rusage_user 2481.696713 STAT rusage_system 2119.289164 STAT curr_connections 10 STAT total_connections 31 STAT connection_structures 30 STAT reserved_fds 20 STAT cmd_get 0 STAT cmd_set 499774196 STAT cmd_flush 0 STAT cmd_touch 0 STAT get_hits 0 STAT get_misses 0 STAT delete_misses 0 STAT delete_hits 0 STAT incr_misses 0 STAT incr_hits 0 STAT decr_misses 0 STAT decr_hits 0 STAT cas_misses 0 STAT cas_hits 0 STAT cas_badval 0 STAT touch_hits 0 STAT touch_misses 0 STAT auth_cmds 0 STAT auth_errors 0 STAT bytes_read 558247776939 STAT bytes_written 3998193568 STAT limit_maxbytes 137437904896 STAT accepting_conns 1 STAT listen_disabled_num 0 STAT threads 4 STAT conn_yields 16659013 STAT hash_power_level 27 STAT hash_bytes 1073741824 STAT hash_is_expanding 0 STAT bytes 135580567504 STAT curr_items 116079253 STAT total_items 499774196 STAT expired_unfetched 0 STAT evicted_unfetched 8864296 STAT evictions 8864296 STAT reclaimed 0 END
