Re: After many hours all outbound connections get stuck in SYN_SENT
Huh? Do you mean a PIX blade in a Cisco switch-router chassis? It would be very useful if you could be less vague about the equipment in use. Right it's a PIX blade in Cisco chassis. The PIX is running ASA version 7.0(6) That depends more on your customers' networking attributes then you are sharing or perhaps even know. Perhaps your customer base is very Window-skewed and you simply aren't seeing any Sack Permitted negotiations for the first 37.999 hours. Or perhaps you've had a network glitch, and all of your connections have done a Selective Ack, which the firewall has trashed, leaving all the connections in a wacko state, not just a few which you haven't noticed. I definitely see SACKs over the course of the 38 hours. I don't have any Windows hosts, they are all running Linux except for a very small number that run a proprietary OS and webserver. If the firewall were to get trashed and hose the currently active connections, I would expect that newly initiated connections would work. The actual failure mode needs a packet trace to determine, but you should be able to do this yourself (or ask your local network engineering staff). If your firewall is trashing the Sack field, then it needs to be fixed. Time to raise a case with the Cisco TAC and ask them directly if your PIX version has bug CSCse14419. You can't expect Sack to work when it's being fed trash, so it is important to make sure that is not happening. I've pinged our dude that handles the PIX stuff to see about getting an upgrade to 7.0(7). I should be able to get a packet trace, but it may take some time. At this point I'm getting a lot of resistance and people here telling me to just turn SACK off and not worry about what is causing this issue, but I'd really like to get to the bottom of it. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
[speculation by network engineer -- not kernel hacker -- follows] The router could be sooo crappy that it drops all packets from TCP streams that have SACK enabled and the client has opened 200+ SACK connections previously... something like that? As far as any third party is concerned the existing TCP connections continue to have negotiated SACK Permitted. Only new connections will not negotiate this. So router crappiness promptly disappearing doesn't seem too likely (a way I could see this happening is if the Linux box sends a Ack for each connection and this clears out Sack datastructures on the third party). But I'd be very surprised if the router is acting as anything more that a network-layer device. It might perhaps have some soft connection state being used for generating accounting records. Being Cisco it's probably a switch-router, so it might carry some per-port hard state for validating source IP addresses and ARPs on each port. The firewall is much more likely to be carrying per-flow Sack state. The Cisco PIX had a bug with SACK handling (CSCse14419, fixed in 7.0(7), 7.1(2.34), 7.2(2.2), 8.0(0.141) but perhaps it has regressed). A simple trace either side of the firewall will show the inconsistency between the TCP sequence number (which gets randomised) and the Sack sequence number (which didn't). You could disable the TCP Sequence Number Randomisation feature and see if the fault reoccurs. You'd probably should also investigate the Linux kernel, especially the size and locks of the components of the Sack data structures and what happens to those data structures after Sack is disabled (presumably the Sack data structure is in some unhappy circumstance, and disabling Sack allows the data to be discarded, magically unclaging the box). In the absence of the reporter wanting to dump the kernel's core, how about a patch to print the Sack datastructure when the command to disable Sack is received by the kernel? Maybe just print the last 16b of the IP address? Best wishes, Glen -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
I still dont understand. tcpdump -p -n -s 1600 -c 1 doesnt reveal User data at all. Without any exact data from you, I am afraid nobody can help. Oh, I didn't see that you specified specific options. I'll still have to anonymize 2000+ IP addresses, but I think there is an open source tool that will do this for you. 2) Are you sure you are not using connection tracking, and hit a limit on it ? I'm using ip_conntrack, but the limit I have for max entries is 65K. The most I've seen in there are a couple thousand- that was one of the first things I monitored very closely. Now please try without conn tracking module. I saw many failures in the past that were trigered by conntrack. Do you have some firewall rules, using some netfilter modules like hashlimit ? I will have to look into this. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
But I'd be very surprised if the router is acting as anything more that a network-layer device. It might perhaps have some soft connection state being used for generating accounting records. Being Cisco it's probably a switch-router, so it might carry some per-port hard state for validating source IP addresses and ARPs on each port. The firewall is much more likely to be carrying per-flow Sack state. The Cisco PIX had a bug with SACK handling (CSCse14419, fixed in 7.0(7), 7.1(2.34), 7.2(2.2), 8.0(0.141) but perhaps it has regressed). A simple trace either side of the firewall will show the inconsistency between the TCP sequence number (which gets randomised) and the Sack sequence number (which didn't). You could disable the TCP Sequence Number Randomisation feature and see if the fault reoccurs. I do have TCP Sequence # Randomization enabled on my router. However, if this was causing an issue, wouldn't it always occur and cause connection issues, not just after 38 hours of correct operation? I can look into turning this off, but I'll likely have to jump through several hoops which will be challenging if I don't have a very clear definitive reason why this is causing this issue. Plus, I've had this problem with at least 2 other sets of network switches over the past 4 years. I'm actually running 7.0(6), which doesn't have the fix you mentioned. If it really is possible that this issue wouldn't always cause problems, but only after hours of succesful operation, then I could probably motivate the upgrade. I can try to setup a trace, but this is a lot of work for other people in my organization, so it will take quite some time. You'd probably should also investigate the Linux kernel, especially the size and locks of the components of the Sack data structures and what happens to those data structures after Sack is disabled (presumably the Sack data structure is in some unhappy circumstance, and disabling Sack allows the data to be discarded, magically unclaging the box). In the absence of the reporter wanting to dump the kernel's core, how about a patch to print the Sack datastructure when the command to disable Sack is received by the kernel? Maybe just print the last 16b of the IP address? Given the fact that I've had this problem for so long, over a variety of networking hardware vendors and colo-facilities, this really sounds good to me. It will be challenging for me to justify a kernel core dump, but a simple patch to dump the Sack data would be do-able. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Thu, 20 Dec 2007, James Nichols wrote: I still dont understand. tcpdump -p -n -s 1600 -c 1 doesnt reveal User data at all. Without any exact data from you, I am afraid nobody can help. Oh, I didn't see that you specified specific options. I'll still have to anonymize 2000+ IP addresses, but I think there is an open source tool that will do this for you. Even a simple for loop in shell can do that. It's not that hard and there's very little need for manual work! Ingrediments: for, cut, grep and sed. -- i. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Thu, 20 Dec 2007, James Nichols wrote: You'd probably should also investigate the Linux kernel, especially the size and locks of the components of the Sack data structures and what happens to those data structures after Sack is disabled (presumably the Sack data structure is in some unhappy circumstance, and disabling Sack allows the data to be discarded, magically unclaging the box). ...Not sure if you want now to invent such structure. Yes, we have per skb -sacked but again in SYN_SENT there are very few things who touch it at all, and they just set it to zero (though it would not even be mandatory for tcp_transmit_skb, IIRC, checked that just couple of days ago due to other things). Another thing is the rx_opt.sack_ok which is just couple flag bits that tell the TCP variant in use (and it's mostly used only after SYN handshake completes). The rest (the actual SACK blocks) is in the ack_skb but again it has very little meaning in SYN_SENT state unless somebody is crazy enough to add SACK blocks to SYN-ACKs :-). In the absence of the reporter wanting to dump the kernel's core, how about a patch to print the Sack datastructure when the command to disable Sack is received by the kernel? Maybe just print the last 16b of the IP address? Given the fact that I've had this problem for so long, over a variety of networking hardware vendors and colo-facilities, this really sounds good to me. It will be challenging for me to justify a kernel core dump, but a simple patch to dump the Sack data would be do-able. If your symptoms really are: SYNs leaving (if they show up in tcpdump, for sure they've left TCP code already) and SYN-ACK not showing up even in something as early as in tcpdump (for sure TCP side code didn't execute at that point yet), there's very little change that Linux' TCP code has some bug in it, only things that do something in such scenario are the SYN generation and retransmitting SYNs (and those are trivially verifiable from tcpdump). -- i. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
James Nichols wrote I still dont understand. tcpdump -p -n -s 1600 -c 1 doesnt reveal User data at all. Without any exact data from you, I am afraid nobody can help. Oh, I didn't see that you specified specific options. I'll still have to anonymize 2000+ IP addresses, but I think there is an open source tool that will do this for you. tcpdump -p -n -s 1600 -c 1 | perl -pe 's/(\d+\.\d+\.\d+\.\d+)/HIDE.THIS.IP.ADDR/g' -justinb -- Justin Banks BakBone Software [EMAIL PROTECTED] -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
I do have TCP Sequence # Randomization enabled on my router. Huh? Do you mean a PIX blade in a Cisco switch-router chassis? It would be very useful if you could be less vague about the equipment in use. However, if this was causing an issue, wouldn't it always occur and cause connection issues, not just after 38 hours of correct operation? That depends more on your customers' networking attributes then you are sharing or perhaps even know. Perhaps your customer base is very Window-skewed and you simply aren't seeing any Sack Permitted negotiations for the first 37.999 hours. Or perhaps you've had a network glitch, and all of your connections have done a Selective Ack, which the firewall has trashed, leaving all the connections in a wacko state, not just a few which you haven't noticed. The actual failure mode needs a packet trace to determine, but you should be able to do this yourself (or ask your local network engineering staff). If your firewall is trashing the Sack field, then it needs to be fixed. Time to raise a case with the Cisco TAC and ask them directly if your PIX version has bug CSCse14419. You can't expect Sack to work when it's being fed trash, so it is important to make sure that is not happening. Cheers, Glen #include network_engineer.h #undef KERNEL_HACKER -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Dec 20 2007 23:05, Ilpo Järvinen wrote: Given the fact that I've had this problem for so long, over a variety of networking hardware vendors and colo-facilities, this really sounds good to me. It will be challenging for me to justify a kernel core dump, but a simple patch to dump the Sack data would be do-able. If your symptoms really are: SYNs leaving (if they show up in tcpdump, for sure they've left TCP code already) and SYN-ACK not showing up even in something as early as in tcpdump (for sure TCP side code didn't execute at that point yet), there's very little change that Linux' TCP code has some bug in it, only things that do something in such scenario are the SYN generation and retransmitting SYNs (and those are trivially verifiable from tcpdump). Take a machine, put two interfaces in it, configure as bridge (br0 over eth0 and eth1 without any assigned ip addresses), put it between end node and the cisco. tcpdump there, which should give an unbiased view wrt. endnode/cisco. Then perhaps, also configure such a network listening bridge on the other side of the cisco, e.g. on the link to the internet and watch that. Compare the two tcpdumpds and see if sack got trashed. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Sun, 16 Dec 2007, James Nichols wrote: I have a Java application that makes a large number of outbound webservice calls over HTTP/TCP. The hosts contacted are a fixed set of about 2000 hosts and a web service call is made to each of them approximately every 5 mintues by a pool of 200 Java threads. Over time, on average a percentage of these hosts are unreachable for one reason or another, usually because they are on wireless cell phone NICs, so there is a persistent count of sockets in the SYN_SENT state in the range of about 60-80. This is fine, as these failed connection attempts eventually time out. However, after approximately 38 hours of operation, all outbound connection attempts get stuck in the SYN_SENT state. It happens instantaneously, where I go from the baseline of about 60-80 sockets in SYN_SENT to a count of 200 (corresponding to the # of java threads that make these calls). When I stop and start the Java application, all the new outbound connections still get stuck in SYN_SENT state. Is it so that they don't timeout at all? You can collect some of their state from /proc/net/tcp (shows at least timers and attempt counters) During this time, I am still able to SSH to the box and run wget to Google, cnn, etc, so the problem appears to be specific to the hosts that I'm accessing via the webservices. Are you sure that you just don't get unlucky at some point of time and all 200 available threads are just temporarily stuck and your application is just very slowly progressing then? For a long time, the only thing that would resolve this was rebooting the entire machine. Once I did this, the outbound connections could be made succesfully. To the very same hosts? Or to another set of hosts? However, very recently when I had once of these incidents I disabled tcp_sack via: echo 0 /proc/sys/net/ipv4/tcp_sack And the problem almost instanteaously resolved itself and outbound connection attempts were succesful. New or the pending ones? I hadn't attempted this before because I assumed that if any of my network equipment or remote hosts had a problem with SACK, that it would never work. Many bugs just are not like that at all... Usually people who coded things had at least some clue :-), so things work almost correctly... In my case, it worked fine for about 38 hours before hitting a wall where no outbound connections could be made. How accurate number? Is the lockup somehow related to daytime cycle? Is there a kernel buffer or some data structure that tcp_sack uses that gets filled up after an extended period of operation? SACK has pretty little meaning in context of SYNs, there's only the sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK. The SACK scoreboard is currently included to the skbs (has been like this for very long time), so no additional data structures should be there because of SACK... How can I debug this problem in the kernel to find out what the root cause is? I emailed linux-kernel and they asked for output of netstat -s, I can get this the next time it occurs- any other usefull data to collect? /proc/net/tcp couple of times in a row, try something something like this: for i in (seq 1 40); do cat /proc/net/tcp; echo -; sleep 10; done I'm running kernel 2.6.18 on RedHat, but have had this problem occur on earlier kernel versions (all 2.4 and 2.6). I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat has backported them). Though they're not that critical nor anything in them should affect in SYN_SENT state. -- i. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. So... Really unlikely a linux problem, but ... When the problem comes, instead of restarting the application, please take a tcpdump of say 10.000 packets. Then turn off tcp_sack and take a 2nd tcpdump sample, and make both samples available to us. I can take these captures and take a look at the results. Unfortunately, I don't think I'll be able to make the captures available to the general public. I dont understand, why dont you change IPs to mask them with 192.168.X.Y, or just ME, and peer1, peer2, peer... If turning off tcp_sack makes the problem go away, why dont you turn it off all the time ? Unfortunately, I think that will be the answer if I can't get any help fixing this problem in the kernel. It's a bummer, because many of the remote hosts my application communicates with are on wireless links, so there may be performance implications to turning SACK off. Random ideas : 1) Is your server behind a NET router or something ? 2) Are you sure you are not using connection tracking, and hit a limit on it ? -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. When the problem comes, instead of restarting the application, please take a tcpdump of say 10.000 packets. Then turn off tcp_sack and take a 2nd tcpdump sample, and make both samples available to us. I can take these captures and take a look at the results. Unfortunately, I don't think I'll be able to make the captures available to the general public. If turning off tcp_sack makes the problem go away, why dont you turn it off all the time ? Unfortunately, I think that will be the answer if I can't get any help fixing this problem in the kernel. It's a bummer, because many of the remote hosts my application communicates with are on wireless links, so there may be performance implications to turning SACK off. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On 12/19/07, Eric Dumazet [EMAIL PROTECTED] wrote: James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. So... Really unlikely a linux problem, but ... I don't know how you can be so sure. Turning tcp_sack off instantly resovles the problem and all connections are succesful. I can't imagine even the most far-fetched scenario where a router or every single remote endpoints would suddenly stop causing the problem just by removing a single TCP option. I can take these captures and take a look at the results. Unfortunately, I don't think I'll be able to make the captures available to the general public. I dont understand, why dont you change IPs to mask them with 192.168.X.Y, or just ME, and peer1, peer2, peer... I will see if I can do that, but it's major pain with 2000 hosts. Plus, there is application data in the packets that I can't allow into the public domain. I really don't think I can pull it off... I literally would have to go through our legal department. Random ideas : 1) Is your server behind a NET router or something ? What's a NET router? I am behind a Cisco router and a firewall, but these network components have completely been replaced/rebuilt several times in the 4+ years that we've had this problem. I've looked at the logs there and neither are doing anything other than passing the traffic along. 2) Are you sure you are not using connection tracking, and hit a limit on it ? I'm using ip_conntrack, but the limit I have for max entries is 65K. The most I've seen in there are a couple thousand- that was one of the first things I monitored very closely. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Dec 19 2007 12:43, James Nichols wrote: On 12/19/07, Eric Dumazet [EMAIL PROTECTED] wrote: James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. So... Really unlikely a linux problem, but ... I don't know how you can be so sure. Turning tcp_sack off instantly resovles the problem and all connections are succesful. I can't imagine even the most far-fetched scenario where a router or every single remote endpoints would suddenly stop causing the problem just by removing a single TCP option. The router could be sooo crappy that it drops all packets from TCP streams that have SACK enabled and the client has opened 200+ SACK connections previously... something like that? -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
When I stop and start the Java application, all the new outbound connections still get stuck in SYN_SENT state. Is it so that they don't timeout at all? You can collect some of their state from /proc/net/tcp (shows at least timers and attempt counters) The outbound connections to timeout. I've watched that they send tcp_syn_retries SYN packets before eventually timing out. Are you sure that you just don't get unlucky at some point of time and all 200 available threads are just temporarily stuck and your application is just very slowly progressing then? Yeah, I'm sure that it isn't an unlucky point of time. If I restart the application when this problem occurs, all the outbound connections still fail. For a long time, the only thing that would resolve this was rebooting the entire machine. Once I did this, the outbound connections could be made succesfully. To the very same hosts? Or to another set of hosts? Yes, to the same exact set of hosts. And the problem almost instanteaously resolved itself and outbound connection attempts were succesful. New or the pending ones? I'm fairly sure that sockets that were already open in SYN_SENT state when I turned tcp_sack off started to work as the count of sockets in SYN_SENT state drops very rapidly. In my case, it worked fine for about 38 hours before hitting a wall where no outbound connections could be made. How accurate number? Is the lockup somehow related to daytime cycle? It is 38 hours +/- a half hour or so. It isn't related to the time of day, as it happens through out day/night depending on when the server was restarted. A new developement in this area is that after the first 38 hours of system time the problem would occur, so I disable tcp_sack and the problem clears itself up and outbound connections are succesful. After a couple of hours I re-enable tcp_sack and the next SYN_SENT issue doesn't occur until more than 50 hours later (so like 90 hours after system start). It's as if the first time it occurs and I turn tcp_sack off, it doesn't just reset the clock another 38 hours, but gives even more time until the problem occurs again. Is there a kernel buffer or some data structure that tcp_sack uses that gets filled up after an extended period of operation? SACK has pretty little meaning in context of SYNs, there's only the sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK. The SACK scoreboard is currently included to the skbs (has been like this for very long time), so no additional data structures should be there because of SACK... I've been seeing this problem for about 4 years, so could it be related to the scoreboard implementation somehow? /proc/net/tcp couple of times in a row, try something something like this: for i in (seq 1 40); do cat /proc/net/tcp; echo -; sleep 10; done I can set up to do this the next time the problem occurs. I'm running kernel 2.6.18 on RedHat, but have had this problem occur on earlier kernel versions (all 2.4 and 2.6). I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat has backported them). Though they're not that critical nor anything in them should affect in SYN_SENT state. Ok, unless there is direct evidence that there is a fix to this problem in a later kernel I won't be able to upgrade. If there is a redhat provided patch I can probably do that. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
James Nichols a écrit : On 12/19/07, Eric Dumazet [EMAIL PROTECTED] wrote: James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. So... Really unlikely a linux problem, but ... I don't know how you can be so sure. Turning tcp_sack off instantly resovles the problem and all connections are succesful. I can't imagine even the most far-fetched scenario where a router or every single remote endpoints would suddenly stop causing the problem just by removing a single TCP option. I can take these captures and take a look at the results. Unfortunately, I don't think I'll be able to make the captures available to the general public. I dont understand, why dont you change IPs to mask them with 192.168.X.Y, or just ME, and peer1, peer2, peer... I will see if I can do that, but it's major pain with 2000 hosts. Plus, there is application data in the packets that I can't allow into the public domain. I really don't think I can pull it off... I literally would have to go through our legal department. I still dont understand. tcpdump -p -n -s 1600 -c 1 doesnt reveal User data at all. Without any exact data from you, I am afraid nobody can help. Random ideas : 1) Is your server behind a NET router or something ? What's a NET router? I am behind a Cisco router and a firewall, but these network components have completely been replaced/rebuilt several times in the 4+ years that we've had this problem. I've looked at the logs there and neither are doing anything other than passing the traffic along. Typo error, I meant NAT. Most routers doing NAT have some limits, timers, hacks... 2) Are you sure you are not using connection tracking, and hit a limit on it ? I'm using ip_conntrack, but the limit I have for max entries is 65K. The most I've seen in there are a couple thousand- that was one of the first things I monitored very closely. Now please try without conn tracking module. I saw many failures in the past that were trigered by conntrack. Do you have some firewall rules, using some netfilter modules like hashlimit ? -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
The router could be sooo crappy that it drops all packets from TCP streams that have SACK enabled and the client has opened 200+ SACK connections previously... something like that? I don't know, maybe. My router is a fairly new model Cisco and is pretty major (i.e. pretty expensive), so it's not just a total piece of crap. Plus, I never restart it when I see these issues. I just turn tcp_sack off, the problem goes away, and I'm able to renable tcp_sack a few hours later and it works fine until many hours later when I see the SYN_SENT problem again. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Wed, 19 Dec 2007, James Nichols wrote: And the problem almost instanteaously resolved itself and outbound connection attempts were succesful. New or the pending ones? I'm fairly sure that sockets that were already open in SYN_SENT state when I turned tcp_sack off started to work as the count of sockets in SYN_SENT state drops very rapidly. Heh, very rapidly :-/, considering that you have 200 x SYN_SENT flows and if tcp_syn_retries is set to default, you will see something happening quite quickly for sure and the whole situation is over in ~3 mins if I counted correctly. Is there a kernel buffer or some data structure that tcp_sack uses that gets filled up after an extended period of operation? SACK has pretty little meaning in context of SYNs, there's only the sackperm(itted) TCP option which is sent along with the SYN/SYN-ACK. The SACK scoreboard is currently included to the skbs (has been like this for very long time), so no additional data structures should be there because of SACK... I've been seeing this problem for about 4 years, so could it be related to the scoreboard implementation somehow? Scoreboard has no meaning in this context, it is used while _input_ packets are processed. If you don't get SYN-ACKs at all, it doesn't have any meaning. /proc/net/tcp couple of times in a row, try something something like this: for i in (seq 1 40); do cat /proc/net/tcp; echo -; sleep 10; done I can set up to do this the next time the problem occurs. for i in $(seq 1 40); ... is the right way to do the loop. :-) I'm running kernel 2.6.18 on RedHat, but have had this problem occur on earlier kernel versions (all 2.4 and 2.6). I've done some fixes to SACK processing since 2.6.18 (not sure if RedHat has backported them). Though they're not that critical nor anything in them should affect in SYN_SENT state. Ok, unless there is direct evidence that there is a fix to this problem in a later kernel I won't be able to upgrade. If there is a redhat provided patch I can probably do that. ...They won't affect in SYN_SENT. -- i. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Wed, 19 Dec 2007, Eric Dumazet wrote: James Nichols a écrit : On 12/19/07, Eric Dumazet [EMAIL PROTECTED] wrote: James Nichols a écrit : So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), Right, I meant to say SYN+ACK. I don't see them coming back. So... Really unlikely a linux problem, but ... I don't know how you can be so sure. Turning tcp_sack off instantly resovles the problem and all connections are succesful. I can't imagine even the most far-fetched scenario where a router or every single remote endpoints would suddenly stop causing the problem just by removing a single TCP option. You could also check if you can pull same trick off by touching tcp_timestamps. It affects the TCP header as well. -- i.
Re: After many hours all outbound connections get stuck in SYN_SENT
James Nichols a écrit : Well... please dont start a flame war :( Back to your SYN_SENT problem, I suppose the remote IP is known, so you probably could post here the result of a tcdpump ? tcpdump -p -n -s 1600 host IP_of_problematic_peer -c 500 Most probably remote peer received too many attempts from you, and a anti DOS mechanism is droping all SYN packets. Ah well... I remember now that you mentioned tcp_sack setting had an effect, so forget the Most probably... and give some tcpdump traces :) I've run tcpdump for all IPs during this problem. I haven't tried doing it for a single explicit IP address- due to the nature of the workload it's very difficult to know which IPs will be hit at any given moment. What I did see in the full IP captures is that the returning ACKs don't show up in the packet capture. Unfortunately, tcpdump reported that some packets were dropped during the capture. Is it possible that the kernel was dropping the packets before they could be captured by tcpdump? Yes it can happens, because an active sniffer makes the stack using more cpu cycles (timestamping for example). So you see outgoing SYN packets, but no SYN replies coming from the remote peer ? (you mention ACKS, but the first packet received from the remote peer should be a SYN+ACK), client-server SYN server-client SYN+ACK client-server ACK Also, I have some doubts about it being the end points or an intermediate router, please let me know if these are unreasonable: 1) We've completely replaced our routing equipment several times in the past 4 years... totally different colos, router vendors, firewall vendors, firewall rules, etc. 2) It occurs across all remote end points at the exact same time. The endpoints are hetrogenous, run brain-dead OS's that don't do any DOS detection, reboot at random times of the day, are geographically distributed, are on different ISPs, etc. etc. 3) Turning of tcp_sack instantaneously makes the problem go away. If it were endpoints or a router, it seems like a stretch that removing a single TCP option would make the problem instantly resolve itself in so many places other than the originating host. CC to netdev where linux network guys might have an idea. When the problem comes, instead of restarting the application, please take a tcpdump of say 10.000 packets. Then turn off tcp_sack and take a 2nd tcpdump sample, and make both samples available to us. If turning off tcp_sack makes the problem go away, why dont you turn it off all the time ? -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
On Dec 18 2007 21:37, Eric Dumazet wrote: If turning off tcp_sack makes the problem go away, why dont you turn it off all the time ? That would just be workaround. I welcome the efforts to track this; not all users have the time to do so. Disabling tcp_sack also disabled it kernel-wide, which, well... for 2.6.25 there is a TCPOPTSTRIP netfilter target slated with which SACK could be stripped only for a given host list or processes from a UID. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: After many hours all outbound connections get stuck in SYN_SENT
Here is some additional information about this problem as requested. I ran ss -m, but no data was returned, what options should I use with ss to gather relevant information? The output of netstat -s: Ip: 1346453452 total packets received 0 forwarded 0 incoming packets discarded 1345744076 incoming packets delivered 1338284375 requests sent out 50 reassemblies required 15 packets reassembled ok 15 fragments received ok 50 fragments created Icmp: 431 ICMP messages received 0 input ICMP message failed. ICMP input histogram: destination unreachable: 42 echo requests: 6 echo replies: 377 timestamp request: 2 address mask request: 2 747 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 739 echo replies: 6 timestamp replies: 2 Tcp: 13115640 active connections openings 1291131 passive connection openings 381803 failed connection attempts 6445 connection resets received 148 connections established 1339571927 segments received 1330375560 segments send out 2443951 segments retransmited 345 bad segments received. 61292 resets sent Udp: 5608790 packets received 725 packets to unknown port received. 0 packet receive errors 5609766 packets sent TcpExt: 1916 resets received for embryonic SYN_RECV sockets 1290 packets pruned from receive queue because of socket buffer overrun 1250631 TCP sockets finished time wait in fast timer 43568 time wait sockets recycled by time stamp 16323 active connections rejected because of time stamp 262 packets rejects in established connections because of timestamp 18505058 delayed acks sent 3931 delayed acks further delayed because of locked socket Quick ack mode was activated 434830 times 1902 times the listen queue of a socket overflowed 1902 SYNs to LISTEN sockets ignored 1068352581 packets directly queued to recvmsg prequeue. 92424765 packets directly received from backlog 800659035 packets directly received from prequeue 1158417138 packets header predicted 2223869 packets header predicted and directly queued to user 22256941 acknowledgments not containing data received 1109445014 predicted acknowledgments 96 times recovered from packet loss due to fast retransmit 325 times recovered from packet loss due to SACK data 1 bad SACKs received Detected reordering 8 times using FACK Detected reordering 7 times using time stamp 21 congestion windows fully recovered 29 congestion windows partially recovered using Hoe heuristic 452978 congestion windows recovered after partial ack 97 TCP data loss events 2269 timeouts after reno fast retransmit 144 timeouts after SACK recovery 12690 timeouts in loss state 731 fast retransmits 70 forward retransmits 38188 retransmits in slow start 959183 other TCP timeouts TCPRenoRecoveryFail: 67 38 sack retransmits failed 42 times receiver scheduled too late for direct processing 75627 packets collapsed in receive queue due to low socket buffer 6003 DSACKs sent for old packets 13 DSACKs sent for out of order packets 136 DSACKs received 4038 connections reset due to unexpected data 557 connections reset due to early user close 319219 connections aborted due to timeout On 12/16/07, James Nichols [EMAIL PROTECTED] wrote: Hello, I have a Java application that makes a large number of outbound webservice calls over HTTP/TCP. The hosts contacted are a fixed set of about 2000 hosts and a web service call is made to each of them approximately every 5 mintues by a pool of 200 Java threads. Over time, on average a percentage of these hosts are unreachable for one reason or another, usually because they are on wireless cell phone NICs, so there is a persistent count of sockets in the SYN_SENT state in the range of about 60-80. This is fine, as these failed connection attempts eventually time out. However, after approximately 38 hours of operation, all outbound connection attempts get stuck in the SYN_SENT state. It happens instantaneously, where I go from the baseline of about 60-80 sockets in SYN_SENT to a count of 200 (corresponding to the # of java threads that make these calls). When I stop and start the Java application, all the new outbound connections still get stuck in SYN_SENT state. During this time, I am still able to SSH to the box and run wget to Google, cnn, etc, so the problem appears to be specific to the hosts that I'm accessing via the webservices. For a long time, the only thing that would resolve this was rebooting the entire machine. Once I did this, the outbound connections could be made succesfully. However, very recently when I had once of these incidents I disabled tcp_sack via: echo 0 /proc/sys/net/ipv4/tcp_sack
