Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Duane Waddle]

On 8/2/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
>
>
> AIX 5.2 and earlier supported it for UDP (we're getting out of the AIX
> business, so I can't speak to what 5.3 does).  Basically, it would send
> out a
> gratuitous 64K ICMP Echo Request with DF set, and waited to see what came
> back.


AIX 5.3 changes the whole PMTU scheme to work more like the RFCs and not
depend on out of band ICMP packets.

>From http://www.ibm.com/developerworks/aix/library/au-aix5l-me.html :

PMTU discovery in AIX 5L Version 5.3
The current Path Maximum Transmission Unit (PMTU) discovery implementation
uses ICMP Echo Request and ICMP Echo Reply packets to discover PMTU. Some
system administrators set up their firewall to drop ICMP Echo packets,
resulting in the above method of PMTU discovery to fail. The PMTU discovery
mechanism in AIX 5L Version 5.3 is implemented with TCP packets and UDP
datagram instead of ICMP Echo packets.

Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Valdis . Kletnieks]

On Thu, 02 Aug 2007 18:33:16 PDT, Jim Shankland said:

> Hmm; I've never actually heard of anybody doing PMTUD on non-TCP
> traffic, though it's possible.  Does anybody actually do it?

AIX 5.2 and earlier supported it for UDP (we're getting out of the AIX
business, so I can't speak to what 5.3 does).  Basically, it would send out a
gratuitous 64K ICMP Echo Request with DF set, and waited to see what came back.
 I ended up turning it off all over, simply because we didn't have enough
UDP-based services that actually hit frag issues to make a difference.

---
'man no' (Network Options) says:

no { -a | -d Attribute | -o Attribute [ =NewValue ] }

udp_pmtu_discover Enables or disables path MTU discovery for UDP applications.
UDP applications must be specifically written to utilize path MTU discovery. A
value of 0 disables the feature, while a value of 1 enables it. This attribute
only applies to AIX 4.2.1 or later. udp_pmtu_discover is a runtime attribute. In
versions prior to AIX 4.3.3, the default value is 0 (disabled); in AIX 4.3.3 and
later versions, the default value is 1 (enabled).
---

The manpage lies - It has to be specifically written to *benefit from* PMTUD.
It would go ahead and do it, and then 98% of the UDP programs wouldn't change
their behavior.  So all you got was lots of gratuitous ICMP mobygrams.

It *may* have made a small difference during a short window, when NFS-over-TCP
support was still rare, and the 4500-octet FDDI MTU was sometimes to be found.


pgpM2BhhEjM7a.pgp
Description: PGP signature

Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Jim Shankland]

Adrian Chadd wrote:

On Thu, Aug 02, 2007, Jim Shankland wrote:


Linux has a nifty iptables option (clamp-mss-to-pmtu) to rewrite the
MSS in TCP SYN packets when forwarding a packet onto a link with
a lower MTU than the MSS in the packet.  Works like a charm.  If every
packet forwarding device on the Internet did this, PMTUD would not be
needed.  As is, PMTUD is simply broken, due to widespread firewall
misconfiguration.  As in so many other cases of Internet misbehavior,
you can avoid being part of the problem, but you can't be the solution.


.. non-TCP traffic?


Hmm; I've never actually heard of anybody doing PMTUD on non-TCP
traffic, though it's possible.  Does anybody actually do it?

Jim Shankland

Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Adrian Chadd]

On Thu, Aug 02, 2007, Jim Shankland wrote:

> Linux has a nifty iptables option (clamp-mss-to-pmtu) to rewrite the
> MSS in TCP SYN packets when forwarding a packet onto a link with
> a lower MTU than the MSS in the packet.  Works like a charm.  If every
> packet forwarding device on the Internet did this, PMTUD would not be
> needed.  As is, PMTUD is simply broken, due to widespread firewall
> misconfiguration.  As in so many other cases of Internet misbehavior,
> you can avoid being part of the problem, but you can't be the solution.

.. non-TCP traffic?




Adrian

Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Jim Shankland]

Robert Boyle wrote:
Either your firewall/router or the customer's firewall/router is 
blocking PMTUD packets.  I suspect an overzealous firewall admin

> is blocking all icmp.

Which you can't do anything about if the overzealous firewall admin
is at the other end of the connection.  My repeated, first-hand
experience has been that several of the better-known web sites out
there will happily send out 1500-byte packets with DF set, then
ignore the DEST_UNREACH/FRAG_NEEDED icmp responses they get.  If you're
on the client end of this, you're sunk unless you initiate the
connection specifying a lower MSS.

Linux has a nifty iptables option (clamp-mss-to-pmtu) to rewrite the
MSS in TCP SYN packets when forwarding a packet onto a link with
a lower MTU than the MSS in the packet.  Works like a charm.  If every
packet forwarding device on the Internet did this, PMTUD would not be
needed.  As is, PMTUD is simply broken, due to widespread firewall
misconfiguration.  As in so many other cases of Internet misbehavior,
you can avoid being part of the problem, but you can't be the solution.

Jim Shankland

Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Robert Boyle]

At 09:30 AM 8/2/2007, Craig D. Rice wrote:
For four months dozens of our users who are Comcast subscribers have 
had difficulty reaching St. Olaf College's and Carleton College's 
network services.


We have worked through everything we can think of with our Onvoy 
(regional ISP) network engineers. We have isolated the problem a 
couple of Comcast's IP subnets, but need a contact within Comcast to 
further troubleshoot.


(snip)

Either your firewall/router or the customer's firewall/router is 
blocking PMTUD packets. Fragment needed, but don't fragment bit set. 
Look at your ICMP access list and make sure you are allowing: permit 
icmp any any unreachable from any Internet address. I suspect an 
overzealous firewall admin is blocking all icmp. Read the acronym to 
him/her and explain that some icmp is necesary for the Internet to work.


-Robert



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

Re: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[William Herrin]

On 8/2/07, Craig D. Rice <[EMAIL PROTECTED]> wrote:
> We have already attempted the usual troubleshooting and have eliminated user
> problems, computer problems, server problems, cable modem problems, and
> Linksys router problems. Traceroutes have been somewhat inconclusive since
> Onvoy blocks ICMP within its network.

Craig,

This rings a bell. Do they block the mandatory ICMP
fragmentation-needed messages?

Try this: on your web server, reduce the MTU from 1500 bytes to 1400
bytes and see if the affected comcast users can now access your web
server.

Regards,
Bill Herrin



-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

RE: Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Durand, Alain]

I've forwarded your message to the appropriate team within Comcast.

  - Alain. 

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of Craig D. Rice
> Sent: Thursday, August 02, 2007 9:30 AM
> To: nanog@merit.edu
> Subject: Seeking Comcast Contact: need to troubleshoot packet 
> loss and/or asymmetric routing issue between Comcast & Onvoy
> 
> 
> 
> For four months dozens of our users who are Comcast 
> subscribers have had difficulty reaching St. Olaf College's 
> and Carleton College's network services.
> 
> We have worked through everything we can think of with our 
> Onvoy (regional
> ISP) network engineers. We have isolated the problem a couple 
> of Comcast's IP subnets, but need a contact within Comcast to 
> further troubleshoot.
> 
> The behavior in a nutshell:
> 
> --
> 
> User A on Comcast Subnet B browses to www.stolaf.edu (http or 
> https, other web sites on-site and @carleton.edu behave the 
> same). Our access_log shows an initial "GET /" of our 
> homepage, then very slow (if any) subsequent requests (for 
> our stylesheet or homepage images). Ping's look fine; 
> traceroute's look as reasonable. Telnet's to port 80 and 
> other services do seem to respond, albeit very slowly.
> 
> User A has the same problem with access @carleton.edu but can 
> access everything else (including other Onvoy customers) 
> without any trouble whatsoever.
> 
> If User A then removes his Linksys router and connects his 
> computer directly to the cable modem, he acquires an IP 
> address in Comcast Subnet C. Then, everything works fine, 
> including access to www.stolaf.edu and www.carleton.edu. He 
> puts the Linksys router back in (which still has the IP 
> address in Comcast Subnet B), and the problem returns.
> 
> The problem IP subnets are completely consistent.
> 
>  Known WORKING IP Subnets: 75.72.0.0, 24.x Known 
> NON-WORKING IP Subnets: 71.x, 73.x
> 
> --
> 
> We have already attempted the usual troubleshooting and have 
> eliminated user problems, computer problems, server problems, 
> cable modem problems, and Linksys router problems. 
> Traceroutes have been somewhat inconclusive since Onvoy 
> blocks ICMP within its network.
> 
> So, why just St. Olaf and Carleton services? We are on a 
> shared physical link from Onvoy, though on different VLANs. 
> Onvoy has verified everything they can (routing, packet loss, 
> etc.) between them and us, and I'm not sure what additional 
> questions I can ask of them to test. Suggestions?
> 
> Maybe Comcast has a broken transparent proxy on part(s) of 
> their network? 
> But they have told us they have nothing like this anywhere on 
> their network.
> 
> Maybe there is some asymmetric routing somewhere, though all 
> the investigation there has come up empty.
> 
> A third possibility is some kind of packet loss, but there is 
> little if any evidence of that.
> 
> So, we are really at a loss and seek any suggestions you all 
> might have. And a contact in Comcast network engineering 
> would be especially useful to continue our troubleshooting.
> 
> With thanks,
> Craig
> -- 
> Craig D. Rice  Associate Director of 
> Information Systems
> [EMAIL PROTECTED]Information and Instructional 
> Technologies
> +1 507 786-3631 St. 
> Olaf College
> +1 507 786-3096 FAX 1510 St. 
> Olaf Avenue
> http://www.stolaf.edu/people/cdr Northfield, MN  
> 55057-1097  USA
> 

Seeking Comcast Contact: need to troubleshoot packet loss and/or asymmetric routing issue between Comcast & Onvoy

[Craig D. Rice]

For four months dozens of our users who are Comcast subscribers have had 
difficulty reaching St. Olaf College's and Carleton College's network services.


We have worked through everything we can think of with our Onvoy (regional 
ISP) network engineers. We have isolated the problem a couple of Comcast's 
IP subnets, but need a contact within Comcast to further troubleshoot.


The behavior in a nutshell:

--

User A on Comcast Subnet B browses to www.stolaf.edu (http or https, other 
web sites on-site and @carleton.edu behave the same). Our access_log shows 
an initial "GET /" of our homepage, then very slow (if any) subsequent 
requests (for our stylesheet or homepage images). Ping's look fine; 
traceroute's look as reasonable. Telnet's to port 80 and other services do 
seem to respond, albeit very slowly.


User A has the same problem with access @carleton.edu but can access 
everything else (including other Onvoy customers) without any trouble 
whatsoever.


If User A then removes his Linksys router and connects his computer directly 
to the cable modem, he acquires an IP address in Comcast Subnet C. Then, 
everything works fine, including access to www.stolaf.edu and 
www.carleton.edu. He puts the Linksys router back in (which still has the IP 
address in Comcast Subnet B), and the problem returns.


The problem IP subnets are completely consistent.

Known WORKING IP Subnets: 75.72.0.0, 24.x
Known NON-WORKING IP Subnets: 71.x, 73.x

--

We have already attempted the usual troubleshooting and have eliminated user 
problems, computer problems, server problems, cable modem problems, and 
Linksys router problems. Traceroutes have been somewhat inconclusive since 
Onvoy blocks ICMP within its network.


So, why just St. Olaf and Carleton services? We are on a shared physical 
link from Onvoy, though on different VLANs. Onvoy has verified everything 
they can (routing, packet loss, etc.) between them and us, and I'm not sure 
what additional questions I can ask of them to test. Suggestions?


Maybe Comcast has a broken transparent proxy on part(s) of their network? 
But they have told us they have nothing like this anywhere on their network.


Maybe there is some asymmetric routing somewhere, though all the 
investigation there has come up empty.


A third possibility is some kind of packet loss, but there is little if any 
evidence of that.


So, we are really at a loss and seek any suggestions you all might have. And 
a contact in Comcast network engineering would be especially useful to 
continue our troubleshooting.


With thanks,
Craig
--
Craig D. Rice  Associate Director of Information Systems
[EMAIL PROTECTED]Information and Instructional Technologies
+1 507 786-3631 St. Olaf College
+1 507 786-3096 FAX 1510 St. Olaf Avenue
http://www.stolaf.edu/people/cdr Northfield, MN  55057-1097  USA