Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread dormando
inline responses. also, I dunno if I missed it but what version were you on originally? are the start arguments the same? On Sat, 13 Oct 2018, Jim Jones wrote: > The "-n" and "ext_item_size" options were picked to help handle the large > volume of small key values our service generates.  But if

Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread Jim Jones
The "-n" and "ext_item_size" options were picked to help handle the large volume of small key values our service generates. But if increasing either will prevent potential deadlocks, I'm sure we'd prefer to accept the efficiency hit. The two servers that stopped accepting connections were only

Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread dormando
Sounds like the daemon hardlocked. Some of your start arguments are fairly aggressive (ext_item_size and -n especially), I'll double check that those won't cause problems like this. First, to confirm: these two hung machines were only getting writes the whole time? no reads? Any info you can

Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread Jim Jones
The commandline arguments used are: -u memcached -m 236544 -c 64000 -p 11211 -t 32 -C -n 5 -f 1.05 -o ext_path=/mnt/memcache:1700G,ext_path=/mnt1/memcache:1700G,ext_path=/mnt2/memcache:1700G,ext_path=/mnt3/memcache:1700G,ext_threads=32,ext_item_size=64 And we have some data, but frankly when the