Hi, I did a stupid mistake, the big machine Memcached version seems different. I guess that is the problem.
Thanks, Alireza On Mon, Dec 16, 2019 at 2:16 AM Alireza Sanaee <sarsan...@gmail.com> wrote: > Hi, > > I'm investigating the Linux load balancer, meanwhile, I'm trying to > understand what is happening in Memcached and just noticed the different > number of Memcached threads on my machines. Linux doesn't provide > immediate access to generally latency-sensitive threads/processes like > Memcached worker threads, causing HoL and eventually long-tail latency, not > a new thing though. > > But before that, I should know what worker threads I need to consider. You > gave me some pointers to different internal threads of Memcached though. > Even intermittent occurrence of some internal worker threads of > Memcached(rebalancer, crawler or ...) might block some requests (I'm not > sure if that's the case or not). ULE scheduler sounds suffering from the > same flaw. > > The connection dispatcher is there, but I don't have so many connections > creation, I have a good number of clients, and send requests at some rates > until the end of the experiment. I'm using mutilate as my workload > generator. > > Sure, I can check whether those are idle or not, I'm actually recording > everything from `/proc/<memcached-pid>/..../stat` so the status is also > available there. It is a true fact that internal threads are IDLE most of > the time and it is some sort of visible in the plot, but I just want to > make sure that all mostly idle ones are just internal threads and not > workers. As you said some times workloads are not loaded evenly making the > worker threads more difficult to distinguish. I think this doesn't matter > now. > > The main issue here is that I have 6 threads on my big machine and 10 > threads on my small machine, while I have the same Memcached configuration > for both machines. I have attached the numbers for the two machines. > > Thanks, > Alireza > > On Sun, Dec 15, 2019 at 3:40 PM dormando <dorma...@rydia.net> wrote: > >> What're you trying to accomplish? >> >> Can you include the output of "stats" and "stats settings" on both >> machines? >> >> Dumb question but you've looked at the output of `ps auxH`? If just using >> htop you may not see the threads that're idle. >> >> TCP connections are pinned to a specific worker thread on connection. >> Trivial benchmarks may not load the worker threads evenly, as the >> connections are handed to threads evenly via round robin. >> >> On Sun, 15 Dec 2019, Alireza Sanaee wrote: >> >> > Hi, >> > Thank you for the information, >> > >> > Sorry for miss using the word there, yes that's all threads. I'm using >> the Memcached 1.5.20. I build it myself and then run my >> experiments($MEMCACHED -u >> > root -p 11211 -m $MAXMEM -c 1024 -t $MEMCACHED_THREADS). And I'm >> checking the number of Memcached threads in htop output. It showed me 10 >> threads(workers >> > included) in one machine and 6 threads(workers included) on the other >> one. >> > >> > To share some more information, I have 200GB of memory for the bigger >> machine that creates only 6 threads, and we have only 16GB of memory for >> the machine >> > that creates 10 threads. I'm just thinking maybe because the smaller >> machine has less amount of space, and I'm actually filling in up to 15GB >> then I might >> > have more work to do and creates more threads. >> > >> > According to your information, I should expect at least 5 threads other >> than the main workers. So 10 threads look OK, but how about the bigger >> machine >> > which spawns only 6 threads? >> > >> > I also had difficulties in detecting the worker threads that respond to >> GET/SET requests on my results, I have attached two pictures, one of them >> shows >> > the actual location of each worker on various cores, and the second one >> is showing userspace time spent for each worker. Apparently worker thread >> number >> > 1,2,4 and 5 have spent more time in userspace, so I'm concluding here >> that 1,2,4 and 5 are my actual worker threads, and worker 3 and 6 are just >> internal >> > worker threads of Memcached. Does that make sense to you? >> > >> > Thanks, >> > Alireza >> > >> > >> > On Sun, Dec 15, 2019 at 7:19 AM dormando <dorma...@rydia.net> wrote: >> > What version of memcached is on each machine? >> > >> > memcached doesn't use processes, it's multi-threaded. Different >> versions >> > may have a different number of background threads. In the latest >> version >> > there should be at least: >> > >> > - listener thread (main "process") >> > - N worker threads >> > - hash table maintenance thread >> > - async log thread (for `watch` commands) >> > - LRU maintainer thread >> > - LRU crawler thread >> > - slab rebalancer thread >> > >> > they're all idle unless they need to do work. LRU maintenance >> thread is >> > probably the most active, since it executes LRU maintenance work >> deferred >> > from the worker threads. Older versions have some of these >> threads, but >> > they were not enabled by default until 1.5.0. >> > >> > -Dormando >> > >> > On Sat, 14 Dec 2019, Alireza Sanaee wrote: >> > >> > > Hello, >> > > I'm running Memcached on two different machines with different >> specifications. And I specify the number of worker threads = 4 for both >> > machines. However, >> > > the number of child processes of the Memcached server is >> different on two machines. On one of them, I have 6 Memcached child >> processes, and >> > on the other >> > > server, I have 10 Memcached child processes. I'm curious to >> understand how many children processes Memcached is basically spawning other >> > than the worker >> > > threads, and for what tasks? >> > > >> > > I expect the Memcached to spawn only 4 children processes or a >> certain number of children processes on two machines, however, it seems not >> > true. >> > > >> > > Thanks, >> > > Alireza >> > > >> > > -- >> > > >> > > --- >> > > You received this message because you are subscribed to the >> Google Groups "memcached" group. >> > > To unsubscribe from this group and stop receiving emails from >> it, send an email to memcached+unsubscr...@googlegroups.com. >> > > To view this discussion on the web visit >> > >> https://groups.google.com/d/msgid/memcached/da7f492f-c12d-4763-86fc-d07311d21c5a%40googlegroups.com >> . >> > > >> > > >> > >> > -- >> > >> > --- >> > You received this message because you are subscribed to a topic >> in the Google Groups "memcached" group. >> > To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe. >> > To unsubscribe from this group and all its topics, send an email >> to memcached+unsubscr...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912141516160.3156%40dskull >> . >> > >> > -- >> > >> > --- >> > You received this message because you are subscribed to the Google >> Groups "memcached" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to memcached+unsubscr...@googlegroups.com. >> > To view this discussion on the web visit >> > >> https://groups.google.com/d/msgid/memcached/CAL%3D0poZ-A0yCr%3DLAAoqcC041bEfC9rthdVZRHCVV-f7xUJ-e4g%40mail.gmail.com >> . >> > >> > >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "memcached" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> memcached+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912142337470.3156%40dskull >> . >> > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/CAL%3D0pobwb6VSGLcPONdvrk4nQJAXzk_mypxiZCYmSSTtheyeVA%40mail.gmail.com.
