Re: Memcached instance and thread configuration parameters

2014-08-08 Thread Keith
Hi, I was also poking at the same systems with Claudio and I can say for sure that we did see servers 'recover' (they at least returned to 'normal' cpu load) on their own without restarting. I'd also like to clarify that we were accidentally running 24 threads per server and 3 servers when

Re: Memcached instance and thread configuration parameters

2014-08-07 Thread Claudio Santana
Sorry for the late response. My CPU utilization normally is min 2.5% to 6.5% max. So it's interesting you ask this. The reason why I submitted the 1st question is because I've experienced some random CPU utilization spikes. From this about 6% CPU utilization all of the sudden it spikes to 100%

Re: Memcached instance and thread configuration parameters

2014-08-07 Thread Claudio Santana
Forgot to say I'm running version 1.4.13 libevent 2.0.16-stable On Thu, Aug 7, 2014 at 6:08 PM, Claudio Santana claudio.sant...@gmail.com wrote: Sorry for the late response. My CPU utilization normally is min 2.5% to 6.5% max. So it's interesting you ask this. The reason why I submitted

Re: Memcached instance and thread configuration parameters

2014-08-07 Thread dormando
Please upgrade. If you have problems with the latest version we can look into it more. You can also look at command counters for odd commands being given: make sure nobody's running flushes, or stats sizes, or stats cachedump since those can cause CPU spikes and hangs. With 1.4.20 you can use

Re: Memcached instance and thread configuration parameters

2014-08-07 Thread Claudio Santana
I run every minute stats, stats items and stats slabs. the only commands executed are remove, incr, add, get, set and cas. I'm running now with 6 threads per instance with 3 per server and haven't had the issue again, not that this change fixed it. I'll definitely update. On Aug 7, 2014 6:13

Re: Memcached instance and thread configuration parameters

2014-08-07 Thread dormando
Those three stats commands aren't problematic. The others I listed are. Sadly there aren't stats counters for them, I think... Are you sure it's not completely crashing after the CPU spike? it actually recovers on its own? On Thu, 7 Aug 2014, Claudio Santana wrote: I run every minute stats,

Re: Memcached instance and thread configuration parameters

2014-08-07 Thread dormando
No command can take up much time. If all other commands hang up, it's either a long-running stats command like I listed before, or a hang bug (though I don't know why it would recover on its own). We've fixed a lot of those since .13, so I'd still advocate upgrading at least some instances to see

Memcached instance and thread configuration parameters

2014-08-04 Thread Claudio Santana
I have this Memcached cluster where 3 instances of Memcached run in a single server. These servers have 24 cores, each instance is configured to have 8 threads each. Each individual instance serves have about 5000G gets/sets a day and about 3k current connections. What would be better?

Re: Memcached instance and thread configuration parameters

2014-08-04 Thread dormando
On Mon, 4 Aug 2014, Claudio Santana wrote: I have this Memcached cluster where 3 instances of Memcached run in a single server. These servers have 24 cores, each instance is configured to have 8 threads each. Each individual instance servesĀ  have about 5000G gets/sets a day and about 3k

Re: Memcached instance and thread configuration parameters

2014-08-04 Thread Claudio Santana
I don't have exact metrics per second but per minute 1.1 million sets and 1.8 million gets which translates to On Mon, Aug 4, 2014 at 6:22 PM, dormando dorma...@rydia.net wrote: On Mon, 4 Aug 2014, Claudio Santana wrote: I have this Memcached cluster where 3 instances of Memcached run in a

Re: Memcached instance and thread configuration parameters

2014-08-04 Thread Claudio Santana
Dormando, thanks for the quick response. Sorry for the confusion, I don't have exact metrics per second but per minute 1.12 million sets and 1.8 million gets which translates to 18,666 sets per minute and 30,000 gets per second. These stats are per Memcached instance which I currently run 3 on

Re: Memcached instance and thread configuration parameters

2014-08-04 Thread dormando
You could run one instance with one thread and serve all of that just fine. have you actually looked at graphs of the CPU usage of the host? memcached should be practically idle with load that low. One with -t 6 or -t 8 would do it just fine. On Mon, 4 Aug 2014, Claudio Santana wrote: