Since this list likes to speculate with little facts on a regular basis
(and I'll admit to being as guilty as anyone) I throw this one out for
opinions :
We were seeing very odd behavior on a Cogent circuit following a
software upgrade to tol01.atlas. Two traceroutes:
mark@angola-gw> traceroute 74.125.226.6
traceroute to 74.125.226.6 (74.125.226.6), 30 hops max, 40 byte packets
1 * * gi1-1.ccr01.tol01.atlas.cogentco.com (38.104.148.5) 110.315 ms
2 te4-2.ccr01.sbn01.atlas.cogentco.com (154.54.7.154) 139.520 ms
196.910 ms 5.728 ms
3 * * *
4 * te0-5-0-5.ccr21.ord03.atlas.cogentco.com (154.54.44.174) 8.310
ms te0-0-0-7.ccr21.ord03.atlas.cogentco.com (154.54.25.70) 8.752 ms
5 te0-0-0-0.ccr22.ord03.atlas.cogentco.com (154.54.24.214) 8.983 ms
te0-1-0-0.ccr22.ord03.atlas.cogentco.com (66.28.4.66) 7.948 ms *
6 * * te-9-1.car4.Chicago1.Level3.net (4.68.127.129) 26.127 ms
7 GOOGLE-INC.car4.Chicago1.Level3.net (4.71.100.22) 38.132 ms
25.120 ms *
8 * * 209.85.254.122 (209.85.254.122) 24.539 ms
9 * 72.14.237.130 (72.14.237.130) 26.134 ms 72.14.237.108
(72.14.237.108) 25.021 ms
MPLS Label=666803 CoS=4 TTL=1 S=1
10 216.239.46.161 (216.239.46.161) 31.816 ms 35.702 ms 32.249 ms
11 72.14.233.142 (72.14.233.142) 32.897 ms * *
12 * yyz06s05-in-f6.1e100.net (74.125.226.6) 33.319 ms *
and a ping over the same path:
--- www.l.google.com ping statistics ---
675 packets transmitted, 323 packets received, 52.1% packet loss
round-trip min/avg/max/stddev = 12.834/28.831/129.743/28.987 ms
and at the same time:
mark@angola-gw> traceroute 38.100.128.10
traceroute to 38.100.128.10 (38.100.128.10), 30 hops max, 40 byte packets
1 gi1-1.ccr01.tol01.atlas.cogentco.com (38.104.148.5) 4.445 ms
1.841 ms 1.713 ms
2 te7-7.ccr02.cle04.atlas.cogentco.com (154.54.5.230) 5.318 ms
te3-2.ccr02.cle04.atlas.cogentco.com (154.54.28.86) 4.755 ms
te7-7.ccr02.cle04.atlas.cogentco.com (154.54.5.230) 4.982 ms
3 te4-2.ccr01.pit02.atlas.cogentco.com (154.54.30.10) 7.997 ms
te3-2.ccr01.pit02.atlas.cogentco.com (154.54.30.6) 7.736 ms
te4-2.ccr01.pit02.atlas.cogentco.com (154.54.30.10) 8.177 ms
4 te0-0-0-5.mpd21.dca01.atlas.cogentco.com (154.54.40.81) 17.197 ms
te0-0-0-5.ccr22.dca01.atlas.cogentco.com (154.54.30.230) 16.907 ms
te0-0-0-5.mpd21.dca01.atlas.cogentco.com (154.54.40.81) 17.008 ms
5 te0-1-0-0.mpd22.dca01.atlas.cogentco.com (154.54.2.193) 17.358 ms
te0-0-0-0.mpd22.dca01.atlas.cogentco.com (154.54.31.38) 17.196 ms
te0-1-0-0.mpd22.dca01.atlas.cogentco.com (154.54.2.193) 18.690 ms
6 te4-2.mpd01.iad03.atlas.cogentco.com (154.54.29.122) 17.885 ms *
18.537 ms
7 cogentco.com (38.100.128.10) 17.836 ms !<10> 17.918 ms !<10>
17.833 ms !<10>
--- 38.100.128.10 ping statistics ---
236 packets transmitted, 236 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.717/27.942/128.011/12.236 ms
sh-3.2#
Works perfectly. There is no asymmetric routing in this scenario
(only 1 BGP peer running during this test), and it is not due to traffic
congestion. Initial speculation over the dropped packets in the trace
to 74.125.226.6 was ICMP depriortization. The results are too
consistent for that to make sense (I have dozens of traceroutes to the
same destination - they all appear similar).
I realize there is a long history of Cogent/L3 ugliness but I'm pretty
sure that this issue has nothing to do with that subject.
Traceroutes and pings from the control plane of tol01.atlas sourced from
38.104.148.5 do not show any odd behavior. Inbound traffic (to us) is
not affected by this. Our workaround while resolving this issue was to
change local-pref on the affected prefixes to send traffic out our other
providers.
The issue started after a software upgrade to tol01.atlas and resolved
after a (reported) reboot of tol01.atlas.
The question is: How does a router break in this manner? It appears
to unintentionally be doing something different with traffic based on
the source address, not the destination address. I realize this can
be done intentionally - but that is not the case here (unless somebody
isn't telling me something).
--
Mark Radabaugh
Amplex
m...@amplex.net 419.837.5015
