RE: Help me make sense of these traceroutes please
Thanks to everyone who responded off list and on. Sam Moats On 2013-12-26 11:21, Josephson, Marcus wrote: Start at slide 50: This is documented further by the following Nanog presentation. http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf -Marcus -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Wednesday, December 25, 2013 10:28 AM To: Martin Hotze Cc: nanog@nanog.org Subject: Re: Help me make sense of these traceroutes please On Wed, Dec 25, 2013 at 8:03 AM, Martin Hotze m.ho...@hotze.com wrote: On 2013-12-25 00:16, Sam Moats wrote: ... You are likely seeing the effects of asymmetric routing. . .. or the effect of passing traffic through NSA infrastructure. Ah... NSA. That's probably it. So much for my theory of a Router virtual chassis straddling the atlantic. or the extra kinetic energy carried by the overseas-bound packet took longer for the router to absorb and rebound with an ICMP. But in all seriousness --- what is probably happening here, is the result of extra hops that don't show up in traceroute. MPLS tunnels could well fit the bill. Other things to consider when latency seems sensitive to destination IP --- are preceding device in the traceroute might also have multiple links to the same device; with one link congested and some form of IP-based load sharing, that happens to be the toward-overseas link. SCNR, #m -- -JH
RE: Help me make sense of these traceroutes please
Start at slide 50: This is documented further by the following Nanog presentation. http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf -Marcus -Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Wednesday, December 25, 2013 10:28 AM To: Martin Hotze Cc: nanog@nanog.org Subject: Re: Help me make sense of these traceroutes please On Wed, Dec 25, 2013 at 8:03 AM, Martin Hotze m.ho...@hotze.com wrote: On 2013-12-25 00:16, Sam Moats wrote: ... You are likely seeing the effects of asymmetric routing. . .. or the effect of passing traffic through NSA infrastructure. Ah... NSA. That's probably it. So much for my theory of a Router virtual chassis straddling the atlantic. or the extra kinetic energy carried by the overseas-bound packet took longer for the router to absorb and rebound with an ICMP. But in all seriousness --- what is probably happening here, is the result of extra hops that don't show up in traceroute. MPLS tunnels could well fit the bill. Other things to consider when latency seems sensitive to destination IP --- are preceding device in the traceroute might also have multiple links to the same device; with one link congested and some form of IP-based load sharing, that happens to be the toward-overseas link. SCNR, #m -- -JH
Re: Help me make sense of these traceroutes please
From: Jeroen Massar jer...@massar.ch To: s...@circlenet.us, nanog@nanog.org Subject: Re: Help me make sense of these traceroutes please On 2013-12-25 00:16, Sam Moats wrote: Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing. You are likely seeing the effects of asymmetric routing. . .. or the effect of passing traffic through NSA infrastructure. SCNR, #m
Re: Help me make sense of these traceroutes please
On Wed, Dec 25, 2013 at 8:03 AM, Martin Hotze m.ho...@hotze.com wrote: On 2013-12-25 00:16, Sam Moats wrote: ... You are likely seeing the effects of asymmetric routing. . .. or the effect of passing traffic through NSA infrastructure. Ah... NSA. That's probably it. So much for my theory of a Router virtual chassis straddling the atlantic. or the extra kinetic energy carried by the overseas-bound packet took longer for the router to absorb and rebound with an ICMP. But in all seriousness --- what is probably happening here, is the result of extra hops that don't show up in traceroute. MPLS tunnels could well fit the bill. Other things to consider when latency seems sensitive to destination IP --- are preceding device in the traceroute might also have multiple links to the same device; with one link congested and some form of IP-based load sharing, that happens to be the toward-overseas link. SCNR, #m -- -JH
Re: Help me make sense of these traceroutes please
On Tue, 24 Dec 2013 19:03:02 -0500, Sam Moats said: Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness I once heard the claim that if you couldn't explain your network design and have the listener understand it after you had split a pitcher of Guiness, it was probably too complicated. pgpwmQleyV_4U.pgp Description: PGP signature
Re: Help me make sense of these traceroutes please
with a bunch of IT folks and an ample supply of Guinness. My ex used to call it design fluid. :-) Happy holidays, everyone! Anne Anne P. Mitchell, Attorney at Law CEO/President ISIPP SuretyMail Email Accreditation http://www.ISIPP.com Member, Cal. Bar Cyberspace Law Committee Author: Section 6 of the CAN-SPAM Act of 2003 How do you get to the inbox instead of the spam filter? SuretyMail! Helping businesses keep their email out of the junk folder since 1998 http://www.isipp.com/SuretyMail Author, They're Your Kids Too: The Single Father's Guide to Defending Your Fatherhood in a Broken Family Law System http://www.amazon.com/Theyre-Your-Kids-Too-Fatherhood/dp/061551443X
Re: Help me make sense of these traceroutes please
Pitcher of Guinness!?! What blasphemy is this, the only way to drink it is via individually poured pint glasses. Back to the issues I'd say MPLS or GHCQ before NSA. On 25 Dec 2013 15:52, valdis.kletni...@vt.edu wrote: On Tue, 24 Dec 2013 19:03:02 -0500, Sam Moats said: Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness I once heard the claim that if you couldn't explain your network design and have the listener understand it after you had split a pitcher of Guiness, it was probably too complicated.
Re: Help me make sense of these traceroutes please
Thats why you're a bacon zombie. If you were a living person you'd know free beer tastes the same irrespective of the containment vessel. ;) I hope Santa brought all of you what you wanted. If not, blame UPS. Sent from my Mobile Device. Original message From: Bacon Zombie baconzom...@gmail.com Date: 12/25/2013 11:24 AM (GMT-09:00) To: valdis.kletni...@vt.edu Cc: s...@circlenet.us,nanog@nanog.org Subject: Re: Help me make sense of these traceroutes please Pitcher of Guinness!?! What blasphemy is this, the only way to drink it is via individually poured pint glasses. Back to the issues I'd say MPLS or GHCQ before NSA. On 25 Dec 2013 15:52, valdis.kletni...@vt.edu wrote: On Tue, 24 Dec 2013 19:03:02 -0500, Sam Moats said: Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness I once heard the claim that if you couldn't explain your network design and have the listener understand it after you had split a pitcher of Guiness, it was probably too complicated.
Help me make sense of these traceroutes please
Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing. First some background, I have Level3 circuits in the US and some services in Europe. From Comcast to the US level3 IPs the performance is excellent. The same traceroute to Europe is terrible. The strange part is the problem appears to begin stateside on the same infrastructure that carriers the us traffic. Here is a trace to one of my IPs in the US from Comcast Tracing route to 4.30.x.x over a maximum of 30 hops 1 3 ms 1 ms 1 ms 10.1.1.1 230 ms29 ms29 ms 71.62.150.1 3 9 ms 9 ms 9 ms xe-0-1-0-32767-sur01.winchester.va.richmond.comc ast.net [68.85.71.165] 4 9 ms14 ms10 ms xe-9-0-3-0-ar02.staplesmllrd.va.richmond.comcast .net [68.86.125.149] 532 ms30 ms34 ms 68.86.91.153 636 ms38 ms53 ms 23.30.207.98 734 ms28 ms33 ms vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62] 829 ms28 ms20 ms ae-63-63.ebr3.Atlanta2.Level3.net [4.69.148.241] 927 ms29 ms30 ms ae-2-2.ebr1.Washington1.Level3.net [4.69.132.86] 1024 ms30 ms24 ms ae-71-71.csw2.Washington1.Level3.net [4.69.134.1 34] 1129 ms31 ms39 ms ae-41-90.car1.Washington1.Level3.net [4.69.149.1 95] 1230 ms30 ms29 ms ae-2-23.edge7.Washington1.Level3.net [4.68.106.2 38] 1338 ms44 ms43 ms 4.79.x.x 14 *** Request timed out. (My firewall) 1539 ms39 ms39 ms 4.30.x.x Trace complete. Now here is the same computer tracing to a level3 circuit in Ireland. Tracing route to xxx.yyy.ie [193.1.x.x] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.1.1.1 238 ms33 ms25 ms 71.62.150.1 310 ms 9 ms 9 ms xe-0-1-0-32767-sur01.winchester.va.richmond.comc ast.net [68.85.71.165] 414 ms15 ms15 ms xe-9-0-3-0-ar02.staplesmllrd.va.richmond.comcast .net [68.86.125.149] 528 ms30 ms30 ms 68.86.95.65 637 ms37 ms37 ms 23.30.207.98 7 118 ms* 218 ms vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62] 8 119 ms 218 ms 119 ms ae-63-63.ebr3.Atlanta2.Level3.net [4.69.148.241] 9 221 ms 119 ms 119 ms ae-2-2.ebr1.Washington1.Level3.net [4.69.132.86] 10 118 ms 119 ms 118 ms ae-91-91.csw4.Washington1.Level3.net [4.69.134.1 42] 11 119 ms 119 ms 119 ms ae-92-92.ebr2.Washington1.Level3.net [4.69.134.1 57] 12 117 ms 126 ms 120 ms ae-43-43.ebr2.Paris1.Level3.net [4.69.137.57] 13 128 ms 118 ms 120 ms ae-6-6.car1.Dublin3.Level3.net [4.69.148.53] 14 122 ms 225 ms 124 ms 4.69.148.58 15 124 ms 118 ms 120 ms ae-11-11.car1.Dublin1.Level3.net [4.69.136.93] Notice that the hop from 23.30.207.98 to 4.69.150.62 seems very respectable at around 30ms for US bound traffic. However when I'm tracing from the same Comcast network to an IP that is in Europe the very same hope produces 100ms of latency and about 12% packet loss. Why does this hop treat traffic differently based on it's destination? Is this some weird result of complex asymmetrical routing or something else? I can route around this problem, but it does seem strange and I want to understand it Thanks, Sam Moats
Re: Help me make sense of these traceroutes please
On 2013-12-25 00:16, Sam Moats wrote: Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing. You are likely seeing the effects of asymmetric routing. [..] Tracing route to xxx.yyy.ie [193.1.x.x] www.heanet.ie by chance? :) Though you could use for instance: http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi to do a reverse traceroute, do make sure you force your connectivity to IPv4 as that host will do IPv6 too. (locally nullrouting the destination /128 is the trick I use for 'disabling' IPv6 temporarily). Otherwise the HEANET folks are extremely helpful and clued in, you can always ask them for help with issues. It is the end-of-year though and those Irish folks have lots of really good whiskey, Guinness thus you might have to be patient till the new year. Alternatively, you could use a tool like 'tracepath' or 'mtr' as those reports multiple answers to a response and also check for the TTL on the return packets. Greets, Jeroen
Re: Help me make sense of these traceroutes please
On 2013-12-24 18:55, Jeroen Massar wrote: On 2013-12-25 00:16, Sam Moats wrote: Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing. You are likely seeing the effects of asymmetric routing. That's what I was thinking to. [..] Tracing route to xxx.yyy.ie [193.1.x.x] www.heanet.ie by chance? :) Yes they were the owners of the IP I used for the example case and the heanet folks are actually totally awesome :-) Though you could use for instance: http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi to do a reverse traceroute, do make sure you force your connectivity to IPv4 as that host will do IPv6 too. (locally nullrouting the destination /128 is the trick I use for 'disabling' IPv6 temporarily). Otherwise the HEANET folks are extremely helpful and clued in, you can always ask them for help with issues. It is the end-of-year though and those Irish folks have lots of really good whiskey, Guinness thus you might have to be patient till the new year. Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness Alternatively, you could use a tool like 'tracepath' or 'mtr' as those reports multiple answers to a response and also check for the TTL on the return packets. Greets, Jeroen Thanks, this isn't affecting my service now I've worked around it so it's more a curiosity than anything. It seems really odd to me that the same L3 edge router would route the ICMP unreachable back to me via different paths based on the final destination IP of the of the ICMP echo packet. Well its Christmas eve here and the customers are happy so Guinness seems like the best approach now :-) Thanks and have a good Holiday, Sam Moats
Re: Help me make sense of these traceroutes please
On 25 December 2013 00:03, Sam Moats s...@circlenet.us wrote: On 2013-12-24 18:55, Jeroen Massar wrote: On 2013-12-25 00:16, Sam Moats wrote: Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing. You are likely seeing the effects of asymmetric routing. That's what I was thinking to. [..] Tracing route to xxx.yyy.ie [193.1.x.x] www.heanet.ie by chance? :) Yes they were the owners of the IP I used for the example case and the heanet folks are actually totally awesome :-) Though you could use for instance: http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi to do a reverse traceroute, do make sure you force your connectivity to IPv4 as that host will do IPv6 too. (locally nullrouting the destination /128 is the trick I use for 'disabling' IPv6 temporarily). Otherwise the HEANET folks are extremely helpful and clued in, you can always ask them for help with issues. It is the end-of-year though and those Irish folks have lots of really good whiskey, Guinness thus you might have to be patient till the new year. Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness Alternatively, you could use a tool like 'tracepath' or 'mtr' as those reports multiple answers to a response and also check for the TTL on the return packets. Greets, Jeroen Thanks, this isn't affecting my service now I've worked around it so it's more a curiosity than anything. It seems really odd to me that the same L3 edge router would route the ICMP unreachable back to me via different paths based on the final destination IP of the of the ICMP echo packet. Based on the data you provided, my guess is some kind of MPLS transport (please refer to https://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N45.pdf, pages 46-48). HTH. Well its Christmas eve here and the customers are happy so Guinness seems like the best approach now :-) Thanks and have a good Holiday, Sam Moats
