RE: SMTP Time outs and re-transmissions, multiple [7:64617]
It would be nice if we could always return the last condition before the last action. But I rarely do that on production networks, because of the concerns in solve the problems as fast as possible. Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=64903&t=64617 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: SMTP Time outs and re-transmissions, multiple [7:64617]
The issue has gone away as per my last email. I think that maybe the NIC tuning fixed it. I did replace a squashed UTP cable, but if that was causing problems I would have expected to see some Ethernet faults during my packet sniffing session. I really don't like it when there is no black and white problem/solution. I am just waiting for this to come back and haunt me. Unfortunately, when there is no symptom, it's hard to continue looking for the problem. One benefit of this exercise was finding a couple of un-related mistakes/omissions in a few of the router configs. There is a positive in every negative :) Thanks again, Symon -Original Message- From: Priscilla Oppenheimer [mailto:[EMAIL PROTECTED] Sent: 07 March 2003 19:07 To: [EMAIL PROTECTED] Subject: RE: SMTP Time outs and re-transmissions, multiple [7:64617] Symon Thurlow wrote: > > HI Priscilla, > > Thanks for your response. This problem is driving me nuts. > > Looking at sh int, sh frame pvc etc I can't see any significant errors > or problems. > > We have been using auth for some time. No significant changes, other > than a power down on Friday, problem notice don Monday. I have gone > over the router configs and nothing looks wrong. No routes flapping, > all times on routes are consistent. There are no firewalls or access > lists in this equation. > > From the client side, this is the packet flow: > > 3 way handshake ok > Final ack from the client (WINXP) declares a 62420 byte window size. > SMTP starts, ehlo, auth, mail from, etc all ok. > After "send data..." from the server, client sends an ack, > with Windows size 63795 (I guess the client is thinking at this > point that it has a good fast connection, and the server is > emptying it's buffers quickly?). > > I then see 5 packets of SMTP data at > > ARRRGHHH It's 8.15pm, i'm still at the clients site and my laptop > just blue screened (it has the captures...). yay. > > Right, I then see 5 packets of SMTP data (from client to > server) at 1514 bytes a piece. I then see one packet with smtp data > from client to server at 946 bytes. not sure why.> > Then I see 3 acks from the server to the client, ack'ing the > 3rd, 5th, and get this the NEXT SMTP data packet. TCP uses a forward acknowledgement, so that may be normal. Please remember also that TCP ACKs bytes, not packets. The sequence number identifies the byte number in the stream of bytes. It identifies the byte number of the first byte in the segment being sent. The ACK number indentifies the next expected byte. > It looks like > the packet and ack were recieved at the same time. THe acks all have a > window size of 8760. This tells me that either the recieve buffers are > full or set too small on the server? Usually servers have larger receive windows than that these days, but not necessarily. I've seen a lot that have a similar size, and it shouldn't really matter with SMTP I don't think. To tune TCP's window size you would have to tune the TCP stack in the registry probably. You mentioned tuning the buffers on the NIC. I don't think that helps TCP. I don't think TCP knows about the NIC's buffers. That's not to say that tuning the NIC buffers won't help the problem. It might cause the NIC to drop fewer frames, which would certainly help. > > The above 5 packets and 3 acks happens another 4 times (the email is > about 400 Kbytes), then the re-transmissions start. > > I get 6 retransmissions of the 4th to last SMTP data packet from the > client to the server, then the whole sequence starts all over again > from 3 way handshake. Sounds like the server stopped ACKing or left a hole in what it has ACKed? You'll have to study the ACK numbers carefully. It sure sounds like the server has some problems and is losing packets somewhere, either hardware or software. Maybe you need to replace its NIC? You said you tuned the buffers which might help too. ? No rst or fin or anything. > > On the mail server I can see it logging that it is waiting for the > message body, and times out waiting. > > Perhaps the buffer fills up on the server and it can;t empty it > anymore? In this case I would expect to see some sort of notification > from the server to stop sending (a reduced window size perhpas?) Yes, that's what you should expect if TCP gets behind. More evidence that the problem is lower down, with the NIC perhaps?? > > I'm going to train it home (with my nose in TCP/IP illustrated) then > study the captures some more. Sounds like a good idea. TCP/IP Illustrated is great. > If anyone is interested, I > can send you the captures, they are from Sniffer Pro. If you get stuck, I'll t
RE: SMTP Time outs and re-transmissions, multiple [7:64617]
Symon Thurlow wrote: > > HI Priscilla, > > Thanks for your response. This problem is driving me nuts. > > Looking at sh int, sh frame pvc etc I can't see any significant > errors or problems. > > We have been using auth for some time. No significant changes, > other than a power down on Friday, problem notice don Monday. I > have gone over the router configs and nothing looks wrong. No > routes flapping, all times on routes are consistent. There are > no firewalls or access lists in this equation. > > From the client side, this is the packet flow: > > 3 way handshake ok > Final ack from the client (WINXP) declares a 62420 byte window > size. > SMTP starts, ehlo, auth, mail from, etc all ok. > After "send data..." from the server, client sends an ack, > with Windows size 63795 (I guess the client is thinking at this > point that it has a good fast connection, and the server is > emptying it's buffers quickly?). > > I then see 5 packets of SMTP data at > > ARRRGHHH It's 8.15pm, i'm still at the clients site and my > laptop just blue screened (it has the captures...). yay. > > Right, I then see 5 packets of SMTP data (from client to > server) at 1514 bytes a piece. I then see one packet with smtp > data from client to server at 946 bytes. not sure why.> > Then I see 3 acks from the server to the client, ack'ing the > 3rd, 5th, and get this the NEXT SMTP data packet. TCP uses a forward acknowledgement, so that may be normal. Please remember also that TCP ACKs bytes, not packets. The sequence number identifies the byte number in the stream of bytes. It identifies the byte number of the first byte in the segment being sent. The ACK number indentifies the next expected byte. > It looks like > the packet and ack were recieved at the same time. THe acks all > have a window size of 8760. This tells me that either the > recieve buffers are full or set too small on the server? Usually servers have larger receive windows than that these days, but not necessarily. I've seen a lot that have a similar size, and it shouldn't really matter with SMTP I don't think. To tune TCP's window size you would have to tune the TCP stack in the registry probably. You mentioned tuning the buffers on the NIC. I don't think that helps TCP. I don't think TCP knows about the NIC's buffers. That's not to say that tuning the NIC buffers won't help the problem. It might cause the NIC to drop fewer frames, which would certainly help. > > The above 5 packets and 3 acks happens another 4 times (the > email is about 400 Kbytes), then the re-transmissions start. > > I get 6 retransmissions of the 4th to last SMTP data packet > from the client to the server, then the whole sequence starts > all over again from 3 way handshake. Sounds like the server stopped ACKing or left a hole in what it has ACKed? You'll have to study the ACK numbers carefully. It sure sounds like the server has some problems and is losing packets somewhere, either hardware or software. Maybe you need to replace its NIC? You said you tuned the buffers which might help too. ? No rst or fin or anything. > > On the mail server I can see it logging that it is waiting for > the message body, and times out waiting. > > Perhaps the buffer fills up on the server and it can;t empty it > anymore? In this case I would expect to see some sort of > notification from the server to stop sending (a reduced window > size perhpas?) Yes, that's what you should expect if TCP gets behind. More evidence that the problem is lower down, with the NIC perhaps?? > > I'm going to train it home (with my nose in TCP/IP illustrated) > then study the captures some more. Sounds like a good idea. TCP/IP Illustrated is great. > If anyone is interested, I > can send you the captures, they are from Sniffer Pro. If you get stuck, I'll take a look. I predict you'll figure it out yourself, though. Good luck. Priscilla > > Cheers, > > Symon > > > > > > From: Priscilla Oppenheimer [mailto:[EMAIL PROTECTED] > Sent: Thu 06/03/2003 6:32 PM > To:[EMAIL PROTECTED] > Subject: RE: SMTP Time outs and re-transmissions, multiple TCP > [7:64617] > > > Symon Thurlow wrote: > > > > Hi all, > > > > I am trying to problem solve and issue where I have multiple > > frame sites > > (1750's) all connecting to a site (3640) to send and receive > > email (POP3 > > SMTP). > > > > They are having problems sending SMTP. Looking at a packet > > trace from a > > site trying to send SMTP, I see lots of re-transmissions. > > > > I have checked with the Frame provider and they assure me that > > no > > packets are being dropped, even though there are a lot getting > > marked > > DE. > > > > I have pretty much excluded the server from being an issue. > > > > IN the packet sniff I also see multiple acks with the same > > sequence > > number, one after the other. This does not look right to me. I > > am about > > to de
RE: SMTP Time outs and re-transmissions, multiple [7:64617]
HI Priscilla, Thanks for your response. This problem is driving me nuts. Looking at sh int, sh frame pvc etc I can't see any significant errors or problems. We have been using auth for some time. No significant changes, other than a power down on Friday, problem notice don Monday. I have gone over the router configs and nothing looks wrong. No routes flapping, all times on routes are consistent. There are no firewalls or access lists in this equation. >From the client side, this is the packet flow: 3 way handshake ok Final ack from the client (WINXP) declares a 62420 byte window size. SMTP starts, ehlo, auth, mail from, etc all ok. After "send data..." from the server, client sends an ack, with Windows size 63795 (I guess the client is thinking at this point that it has a good fast connection, and the server is emptying it's buffers quickly?). I then see 5 packets of SMTP data (from client to server) at 1514 bytes a piece. I then see one packet with smtp data from client to server at 946 bytes. Then I see 3 acks from the server to the client, ack'ing the 3rd, 5th, and get this the NEXT SMTP data packet. It looks like the packet and ack were recieved at the same time. THe acks all have a window size of 8760. This tells me that either the recieve buffers are full or set too small on the server? (I have since increased them). The above 5 packets and 3 acks happens another 4 times (the email is about 400 Kbytes), then the re-transmissions start. I get 6 retransmissions of the 4th to last SMTP data packet from the client to the server, then the whole sequence starts all over again from 3 way handshake. No rst or fin or anything. On the mail server I can see it logging that it is waiting for the message body, and times out waiting. Perhaps the buffer fills up on the server and it can't empty it anymore? In this case I would expect to see some sort of notification from the server to stop sending (a reduced window size perhpas?) I'm going to train it home (with my nose in TCP/IP illustrated) then study the captures some more. If anyone is interested, I can send you the captures, they are from Sniffer Pro. Cheers, Symon From:Priscilla Oppenheimer [mailto:[EMAIL PROTECTED] Sent:Thu 06/03/2003 6:32 PM To: [EMAIL PROTECTED] Subject: RE: SMTP Time outs and re-transmissions, multiple TCP [7:64617] Symon Thurlow wrote: > > Hi all, > > I am trying to problem solve and issue where I have multiple > frame sites > (1750's) all connecting to a site (3640) to send and receive > email (POP3 > SMTP). > > They are having problems sending SMTP. Looking at a packet > trace from a > site trying to send SMTP, I see lots of re-transmissions. > > I have checked with the Frame provider and they assure me that > no > packets are being dropped, even though there are a lot getting > marked > DE. > > I have pretty much excluded the server from being an issue. > > IN the packet sniff I also see multiple acks with the same > sequence > number, one after the other. This does not look right to me. I > am about > to delve in to TCP/IP illustrated to find out about it, but > does this > behaviour trigger anything in your minds? Multiple ACKs with the same sequence number are probably just TCP keepalives. The original TCP RFC didn't mention a keepalive process but the Host Requirements RFC does. That's how it works. After some time of silence a host just sends the last ACK over and over again for a couple hours (typical timeout). Is the client side sending these multiple ACKs? If the server were sending them, and you're seeing them on the client side, then you would have some proof that the server can at least get through, but I bet it's the client side sending them? If you're sniffing on the client side and you see it send retransmissions and multiple ACKs, but you don't see much from the server, I would suspect that the server's packets are getting lost. But what would cause SMTP to get lost but not POP3? A firewall misconfiguration? An MTU issue? What is the last packet exchange you see for SMTP? What error do the users see? For SMTP, did you recently start using authentication or any other new features? We recently started requiring our users to use authentication and had lots of strange problems due to the mix of operating systems. Some of them didn't behave correctly at all, although their misbehavior wasn't exactly what you're describing. Can the users still receive mail? I think you implied that they can. POP3 is based on TCP too. Does it still work? If it still works, but SMTP doesn't, that's a huge clue that TCP and the internetwork are fine. If POP has problems too, start bugging the Frame provider more and do some checking yourself with "show int serial?" Are there lots of errors? Actually, before you blame the Frame provider, though, if possible do some sniffing on the server side and make sure the server is really still sending packets and they are
RE: SMTP Time outs and re-transmissions, multiple [7:64617]
Once we have a complicate problem, like this one; the only way to figure out what was happening was using two Sniffers, one at one side and the other at the other side. Doing this we could see what packets were leaving one side and did not getting to the other side. Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=64708&t=64617 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
RE: SMTP Time outs and re-transmissions, multiple [7:64617]
HI Priscilla, Thanks for your response. This problem is driving me nuts. Looking at sh int, sh frame pvc etc I can't see any significant errors or problems. We have been using auth for some time. No significant changes, other than a power down on Friday, problem notice don Monday. I have gone over the router configs and nothing looks wrong. No routes flapping, all times on routes are consistent. There are no firewalls or access lists in this equation. >From the client side, this is the packet flow: 3 way handshake ok Final ack from the client (WINXP) declares a 62420 byte window size. SMTP starts, ehlo, auth, mail from, etc all ok. After "send data..." from the server, client sends an ack, with Windows size 63795 (I guess the client is thinking at this point that it has a good fast connection, and the server is emptying it's buffers quickly?). I then see 5 packets of SMTP data at ARRRGHHH It's 8.15pm, i'm still at the clients site and my laptop just blue screened (it has the captures...). yay. Right, I then see 5 packets of SMTP data (from client to server) at 1514 bytes a piece. I then see one packet with smtp data from client to server at 946 bytes. not sure why. Then I see 3 acks from the server to the client, ack'ing the 3rd, 5th, and get this the NEXT SMTP data packet. It looks like the packet and ack were recieved at the same time. THe acks all have a window size of 8760. This tells me that either the recieve buffers are full or set too small on the server? The above 5 packets and 3 acks happens another 4 times (the email is about 400 Kbytes), then the re-transmissions start. I get 6 retransmissions of the 4th to last SMTP data packet from the client to the server, then the whole sequence starts all over again from 3 way handshake. No rst or fin or anything. On the mail server I can see it logging that it is waiting for the message body, and times out waiting. Perhaps the buffer fills up on the server and it can;t empty it anymore? In this case I would expect to see some sort of notification from the server to stop sending (a reduced window size perhpas?) I'm going to train it home (with my nose in TCP/IP illustrated) then study the captures some more. If anyone is interested, I can send you the captures, they are from Sniffer Pro. Cheers, Symon From:Priscilla Oppenheimer [mailto:[EMAIL PROTECTED] Sent:Thu 06/03/2003 6:32 PM To: [EMAIL PROTECTED] Subject: RE: SMTP Time outs and re-transmissions, multiple TCP [7:64617] Symon Thurlow wrote: > > Hi all, > > I am trying to problem solve and issue where I have multiple > frame sites > (1750's) all connecting to a site (3640) to send and receive > email (POP3 > SMTP). > > They are having problems sending SMTP. Looking at a packet > trace from a > site trying to send SMTP, I see lots of re-transmissions. > > I have checked with the Frame provider and they assure me that > no > packets are being dropped, even though there are a lot getting > marked > DE. > > I have pretty much excluded the server from being an issue. > > IN the packet sniff I also see multiple acks with the same > sequence > number, one after the other. This does not look right to me. I > am about > to delve in to TCP/IP illustrated to find out about it, but > does this > behaviour trigger anything in your minds? Multiple ACKs with the same sequence number are probably just TCP keepalives. The original TCP RFC didn't mention a keepalive process but the Host Requirements RFC does. That's how it works. After some time of silence a host just sends the last ACK over and over again for a couple hours (typical timeout). Is the client side sending these multiple ACKs? If the server were sending them, and you're seeing them on the client side, then you would have some proof that the server can at least get through, but I bet it's the client side sending them? If you're sniffing on the client side and you see it send retransmissions and multiple ACKs, but you don't see much from the server, I would suspect that the server's packets are getting lost. But what would cause SMTP to get lost but not POP3? A firewall misconfiguration? An MTU issue? What is the last packet exchange you see for SMTP? What error do the users see? For SMTP, did you recently start using authentication or any other new features? We recently started requiring our users to use authentication and had lots of strange problems due to the mix of operating systems. Some of them didn't behave correctly at all, although their misbehavior wasn't exactly what you're describing. Can the users still receive mail? I think you implied that they can. POP3 is based on TCP too. Does it still work? If it still works, but SMTP doesn't, that's a huge clue that TCP and the internetwork are fine. If POP has problems too, start bugging the Frame provider more and do some checking yourself with "show int serial?
