Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread dormando
inline responses. also, I dunno if I missed it but what version were you on originally? are the start arguments the same? On Sat, 13 Oct 2018, Jim Jones wrote: > The "-n" and "ext_item_size" options were picked to help handle the large > volume of small key values our service generates.  But if

Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread Jim Jones
The "-n" and "ext_item_size" options were picked to help handle the large volume of small key values our service generates. But if increasing either will prevent potential deadlocks, I'm sure we'd prefer to accept the efficiency hit. The two servers that stopped accepting connections were only

Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread dormando
Sounds like the daemon hardlocked. Some of your start arguments are fairly aggressive (ext_item_size and -n especially), I'll double check that those won't cause problems like this. First, to confirm: these two hung machines were only getting writes the whole time? no reads? Any info you can

Re: Thousands of CLOSE_WAIT connections

2018-10-13 Thread Jim Jones
The commandline arguments used are: -u memcached -m 236544 -c 64000 -p 11211 -t 32 -C -n 5 -f 1.05 -o ext_path=/mnt/memcache:1700G,ext_path=/mnt1/memcache:1700G,ext_path=/mnt2/memcache:1700G,ext_path=/mnt3/memcache:1700G,ext_threads=32,ext_item_size=64 And we have some data, but frankly when the

Re: Thousands of CLOSE_WAIT connections

2018-10-12 Thread dormando
Hey, Probably unrelated to the original thread. I'm highly interested in understanding what's going on here though, and would appreciate any help you can spare. What data did you collect? The more information you can share the better: 1) start arguments for the daemon 2) stats output (stats,

Re: Thousands of CLOSE_WAIT connections

2018-10-12 Thread jjones
We're testing verison 1.5.10 as a possible upgrade candidate for out older memcached servers, using a pool 9 servers. They are running in parallel with the production pool, also 9 servers. For the test all read requests are going to the production pool, and all updates (set, delete, etc...)

Re: Thousands of CLOSE_WAIT connections

2018-04-22 Thread mr . jiaqimin
Hello, i had the same problem with you . Server OS :Windows Server 2008 memcached 1.4.15 there are many CLOSE_WAIT.and memcached dosent work. did you resoleved this issue? Thanks. 在 2013年9月30日星期一 UTC+8下午11:36:18,Nic Jansma写道: > > Using memcached 1.4.15 > > Over the past few months I've been

Re: Thousands of CLOSE_WAIT connections

2013-09-30 Thread Les Mikesell
On Mon, Sep 30, 2013 at 10:36 AM, Nic Jansma nicjan...@gmail.com wrote: ... The rest of the connections are in FIN_WAIT1, ESTABLISHED, etc, but the vast majority are the ~4,000 CLOSE_WAIT connections. I'm going to try bumping up the connection limit to 32k to avoid this for now, but clearly

Re: Thousands of CLOSE_WAIT connections

2013-09-30 Thread Nic Jansma
On Monday, September 30, 2013 12:23:42 PM UTC-4, LesMikesell wrote: That sounds like your client is opening persistent connections but not reusing them. Doesn't the CLOSE_WAIT state indicate that the client has sent a FIN and yet memcached hasn't internally closed/cleaned up the