Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-21 Thread James Nichols
> Huh? Do you mean a PIX blade in a Cisco switch-router chassis? It > would be very useful if you could be less vague about the > equipment in use. Right it's a PIX blade in Cisco chassis. The PIX is running ASA version 7.0(6) > That depends more on your customers' networking attributes > the

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread Jan Engelhardt
On Dec 20 2007 23:05, Ilpo Järvinen wrote: >> >> Given the fact that I've had this problem for so long, over a variety >> of networking hardware vendors and colo-facilities, this really sounds >> good to me. It will be challenging for me to justify a kernel core >> dump, but a simple patch to du

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread Glen Turner
> I do have TCP Sequence # Randomization enabled on my router. Huh? Do you mean a PIX blade in a Cisco switch-router chassis? It would be very useful if you could be less vague about the equipment in use. > However, > if this was causing an issue, wouldn't it always occur and cause > connectio

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread Justin Banks
James Nichols wrote > > I still dont understand. > > > > "tcpdump -p -n -s 1600 -c 1" doesnt reveal User data at all. > > > > Without any exact data from you, I am afraid nobody can help. > > Oh, I didn't see that you specified specific options. I'll still have > to anonymize 2000+ IP address

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread Ilpo Järvinen
On Thu, 20 Dec 2007, James Nichols wrote: > > You'd probably should also investigate the Linux kernel, > > especially the size and locks of the components of the Sack data > > structures and what happens to those data structures after Sack is > > disabled (presumably the Sack data structure is in

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread Ilpo Järvinen
On Thu, 20 Dec 2007, James Nichols wrote: > > I still dont understand. > > > > "tcpdump -p -n -s 1600 -c 1" doesnt reveal User data at all. > > > > Without any exact data from you, I am afraid nobody can help. > > Oh, I didn't see that you specified specific options. I'll still have > to ano

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread James Nichols
> But I'd be very surprised if the router is acting as anything more > that a network-layer device. It might perhaps have some soft connection > state being used for generating accounting records. Being Cisco > it's probably a switch-router, so it might carry some per-port hard > state for validat

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread James Nichols
> I still dont understand. > > "tcpdump -p -n -s 1600 -c 1" doesnt reveal User data at all. > > Without any exact data from you, I am afraid nobody can help. Oh, I didn't see that you specified specific options. I'll still have to anonymize 2000+ IP addresses, but I think there is an open sou

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-20 Thread Glen Turner
[speculation by network engineer -- not kernel hacker -- follows] > The router could be sooo crappy that it drops all packets from > TCP streams that have SACK enabled and the client has opened > 200+ SACK connections previously... something like that? As far as any third party is concerned the e

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread Ilpo Järvinen
On Wed, 19 Dec 2007, Eric Dumazet wrote: > James Nichols a écrit : > > On 12/19/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: > > > James Nichols a écrit : > > > > > So you see outgoing SYN packets, but no SYN replies coming from the > > > > > remote > > > > > peer ? (you mention ACKS, but the firs

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread Ilpo Järvinen
On Wed, 19 Dec 2007, James Nichols wrote: > > > And the problem almost instanteaously resolved itself and outbound > > > connection attempts were succesful. > > > > New or the pending ones? > > I'm fairly sure that sockets that were already open in SYN_SENT state > when I turned tcp_sack off star

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread James Nichols
> The router could be sooo crappy that it drops all packets from > TCP streams that have SACK enabled and the client has opened > 200+ SACK connections previously... something like that? I don't know, maybe. My router is a fairly new model Cisco and is pretty major (i.e. pretty expensive), so it'

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread Eric Dumazet
James Nichols a écrit : On 12/19/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread Jan Engelhardt
On Dec 19 2007 12:43, James Nichols wrote: >On 12/19/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: >> James Nichols a écrit : >> >> So you see outgoing SYN packets, but no SYN replies coming from the remote >> >> peer ? (you mention ACKS, but the first packet received from the remote >> >> peer sh

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread James Nichols
On 12/19/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: > James Nichols a écrit : > >> So you see outgoing SYN packets, but no SYN replies coming from the remote > >> peer ? (you mention ACKS, but the first packet received from the remote > >> peer should be a SYN+ACK), > > > > Right, I meant to say

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread James Nichols
> > When I stop and start the Java application, all the new outbound > > connections still get stuck in SYN_SENT state. > > Is it so that they don't timeout at all? You can collect some of their > state from /proc/net/tcp (shows at least timers and attempt counters) The outbound connections to

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread James Nichols
> So you see outgoing SYN packets, but no SYN replies coming from the remote > peer ? (you mention ACKS, but the first packet received from the remote > peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. > When the problem comes, instead of restarting the

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread Eric Dumazet
James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. So... Really unlikely a linux p

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-19 Thread Ilpo Järvinen
On Sun, 16 Dec 2007, James Nichols wrote: > I have a Java application that makes a large number of outbound > webservice calls over HTTP/TCP. The hosts contacted are a fixed set > of about 2000 hosts and a web service call is made to each of them > approximately every 5 mintues by a pool of 200 J

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-18 Thread Jan Engelhardt
On Dec 18 2007 21:37, Eric Dumazet wrote: > > If turning off tcp_sack makes the problem go away, why dont you > turn it off all the time ? > That would just be workaround. I welcome the efforts to track this; not all users have the time to do so. Disabling tcp_sack also disabled it kernel-wide, wh

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-18 Thread Eric Dumazet
James Nichols a écrit : Well... please dont start a flame war :( Back to your SYN_SENT problem, I suppose the remote IP is known, so you probably could post here the result of a tcdpump ? tcpdump -p -n -s 1600 host IP_of_problematic_peer -c 500 Most probably remote peer received too many attem

Re: After many hours all outbound connections get stuck in SYN_SENT

2007-12-17 Thread James Nichols
Here is some additional information about this problem as requested. I ran ss -m, but no data was returned, what options should I use with ss to gather relevant information? The output of netstat -s: Ip: 1346453452 total packets received 0 forwarded 0 incoming packets discarded 13

After many hours all outbound connections get stuck in SYN_SENT

2007-12-16 Thread James Nichols
Hello, I have a Java application that makes a large number of outbound webservice calls over HTTP/TCP. The hosts contacted are a fixed set of about 2000 hosts and a web service call is made to each of them approximately every 5 mintues by a pool of 200 Java threads. Over time, on average a perce