Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-08-11 Thread Ronan Kerambrun
Hi all, I have the same problem, but using Varnish. I dont think it is an application bug because Poul-Henning Kamp (the developer of the application) is also a freebsd kernel developper.. # uname -a FreeBSD cache2 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #0: Thu Jun 19 20:37:34 CEST 2008

Re: Probably not a kernel bug (was: Re: FreeBSD 7.0: sockets stuck in CLOSED state...)

2008-06-29 Thread Ali Niknam
Hi Guys, Another public follow-up: Ali has been sending me debugging information privately due to the inclusion of application source code and IP addresses. Tracing of the application suggests that there is an application concurrency bug leading to one socket to be closed twice and another

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Ali Niknam
Hi Eygene, Just a quick me too message: I also used to see this on my 7.x machines. This was with Apache servers in the proxy setup: one I'm wondering: where these 32 bit, or 64 bit machines? I had already tried to debug this situation and Mike Silbersack helped me with patching the

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Robert Watson
On Thu, 26 Jun 2008, Robert Watson wrote: I think the first logical step is to wait for the application to get into that state again, and then run procstat or fstat to dump the file descriptor away for the process. Presumably in the normal steady state, you expect to see a few IPC sockets

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Eygene Ryabinkin
Ali, good day. Fri, Jun 27, 2008 at 08:49:20AM +0200, Ali Niknam wrote: Just a quick me too message: I also used to see this on my 7.x machines. This was with Apache servers in the proxy setup: one I'm wondering: where these 32 bit, or 64 bit machines? amd64. I had already tried to

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Paul
I have the same 'problem' if that helps any.. Sockets stuck for over a month in CLOSED and they have a * for the port on the source IP. tcp4 0 0 67.1.1.1.* 67.1.1.2.1261 CLOSED 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49 EDT 2008 amd64 Doesn't seem

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Eygene Ryabinkin
Paul, good day. Fri, Jun 27, 2008 at 08:45:50AM -0400, Paul wrote: I have the same 'problem' if that helps any.. Sockets stuck for over a month in CLOSED and they have a * for the port on the source IP. tcp4 0 0 67.1.1.1.* 67.1.1.2.1261 CLOSED 7.0-RELEASE-p1 FreeBSD

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Paul
Hi Eygene.. It happens with telnet :) A lot of my closed entries are from telnet so I can't really put a finger on any specific application :/ Eygene Ryabinkin wrote: Paul, good day. Fri, Jun 27, 2008 at 08:45:50AM -0400, Paul wrote: I have the same 'problem' if that helps any.. Sockets

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Robert Watson
On Fri, 27 Jun 2008, Eygene Ryabinkin wrote: Paul, good day. Fri, Jun 27, 2008 at 08:45:50AM -0400, Paul wrote: I have the same 'problem' if that helps any.. Sockets stuck for over a month in CLOSED and they have a * for the port on the source IP. tcp4 0 0 67.1.1.1.* 67.1.1.2.1261 CLOSED

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Robert Watson
On Fri, 27 Jun 2008, Paul wrote: I have the same 'problem' if that helps any.. Sockets stuck for over a month in CLOSED and they have a * for the port on the source IP. tcp4 0 0 67.1.1.1.* 67.1.1.2.1261 CLOSED 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49 EDT 2008 amd64

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-27 Thread Mike Silbersack
On Thu, 26 Jun 2008, Eygene Ryabinkin wrote: I had already tried to debug this situation and Mike Silbersack helped me with patching the kernel. At that days his patches (that went into 7.x) helped, but with higher request rate to the Apache, they seem to be back again. I can try to setup

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-26 Thread Robert Watson
On Wed, 25 Jun 2008, Ali Niknam wrote: precisely matches that what you'd expect: lots of TCP connections in the CLOSED state reflecting a series of connections built by an application but then not properly discarded. Likewise, when the application is killed, all of the connections go away --

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-26 Thread Eygene Ryabinkin
Good day. Wed, Jun 25, 2008 at 07:43:12PM +0200, Ali Niknam wrote: Recently i've been upgrading some of my machines from FreeBSD 6.x amd64 to FreeBSD 7.0 amd64. After upgrading I noticed a weird error/bug. It seems that after several thousand TCP connections some seem to hang in 'CLOSED'

FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Ali Niknam
Dear All, Recently i've been upgrading some of my machines from FreeBSD 6.x amd64 to FreeBSD 7.0 amd64. After upgrading I noticed a weird error/bug. It seems that after several thousand TCP connections some seem to hang in 'CLOSED' state. netstat -n gives: ... tcp4 0 0

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Ali Niknam
This looks like an issue we used to have at work, where a streaming application suddenly started getting kevents for sockets that had been already closed. While that was happening, a netstat output looked just like yours. We never tracked it down, as we moved to other projects :( Was that

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Vlad GALU
On 6/25/08, Ali Niknam [EMAIL PROTECTED] wrote: Dear All, Recently i've been upgrading some of my machines from FreeBSD 6.x amd64 to FreeBSD 7.0 amd64. After upgrading I noticed a weird error/bug. It seems that after several thousand TCP connections some seem to hang in 'CLOSED' state.

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Vlad GALU
On 6/25/08, Ali Niknam [EMAIL PROTECTED] wrote: This looks like an issue we used to have at work, where a streaming application suddenly started getting kevents for sockets that had been already closed. While that was happening, a netstat output looked just like yours. We never tracked

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Robert Watson
On Wed, 25 Jun 2008, Ali Niknam wrote: Recently i've been upgrading some of my machines from FreeBSD 6.x amd64 to FreeBSD 7.0 amd64. After upgrading I noticed a weird error/bug. It seems that after several thousand TCP connections some seem to hang in 'CLOSED' state. Sounds like there's a

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Ali Niknam
Hi Robert, Sounds like there's a bug somewhere. Before we start trying to track it [...] So, with that introduction, we're interested in resolving: Quite comprehensive indeed; thank you for all that information. I was not aware that there was a decoupling between the various parts of the

Re: FreeBSD 7.0: sockets stuck in CLOSED state...

2008-06-25 Thread Lawrence Stewart
Ali Niknam wrote: Hi Robert, [snip] I will double check this once more, but honestly, i strongly doubt it... Also one other thing that I've noticed, is that it's always the input buffer that has bytes left; never the output buffer... Moreover, i've seen that close() reports EBADF, but