Kurt, thanks very much for this. I haven't got time to look at this right away as I am studying for an Exchange 2007 exam tomorrow, but I wanted to say thanks for sharing this. This is what we're all signed up here for, sharing knowledge. Soon we will all have strong traceroute-fu..!
Regards, Andrew 2009/11/13 Kurt Buff <kurt.b...@gmail.com> > An excellent post, and well worth the understanding. > > I am a member of SAGE and USENIX, BTW, and highly recommend the orgs, > even if you can't attend their functions. > > Kurt > > ---------- Forwarded message ---------- > From: Brent Chapman <> > Date: Fri, Nov 13, 2009 at 10:56 > Subject: [SAGE] So you think you know traceroute... > To: sage-memb...@sage.org > > > Most network engineers and sysadmins would probably say that they're > intimately familiar with 'traceroute< > http://en.wikipedia.org/wiki/Traceroute>', > and consider it one of their fundamental network troubleshooting tools... I > certainly do. But you might be amazed to learn, as I did, how much you * > don't* know about traceroute. > > Richard Steenbergen of nLayer Communications, Inc., did an *excellent* > presentation > on traceroute< > http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf > > > at > this month's NANOG <http://www.nanog.org/> (North American Network > Operators > Group) meeting: > > - A Practical Guide to (Correctly) Troubleshooting with > Traceroute< > http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf > >, > by Richard A. > Steenbergen<http://www.linkedin.com/pub/richard-steenbergen/0/8b/398> > of nLayer Communications <http://www.nlayer.net/>. > > Among other things, this presentation shows you: > > - How traceroute works > - What you can learn from the DNS hostnames returned by traceroute > - Where the ISP/carrier boundaries are > - Where the equipment is located, geographically (do you know what a > CLLI code is?) > - What type of equipment the ISP/carrier is using > - What the round trip times reported by traceroute *really* mean > - How you can be led astray by ICMP prioritization, rate limiting, > asymmetric paths, and load balancing > > One of the coolest tricks I learned from this presentation is, to find out > more about what's at the other end of some hop that appears to be a > point-to-point link, assume that the IP address you see is one of the two > addresses in a /30 subnet (as is commonly assigned to point-to-point > links), > and do a DNS reverse lookup of the other address in the /30. > > This is useful, for example, in figuring out which egress port a packet > went > out on, since traceroute normally only shows you the ingress ports for each > device along the way. For example, let's say I was looking at the following > traceroute output, and wanted to know the egress port on router #3, as the > packet moved to router #4: > > *brent%* traceroute www.google.com > traceroute: Warning: www.google.com has multiple addresses; using > 208.67.219.230 > traceroute to google.navigation.opendns.com (208.67.219.230), 64 hops > max, 40 byte packets > 1 192.168.0.1 (192.168.0.1) 3.145 ms 2.573 ms 2.382 ms > 2 75-101-29-1.dsl.static.sonic.net (75.101.29.1) 9.555 ms 9.054 ms > 9.089 ms > 3 127.at-X-X-X.gw3.200p-sf.sonic.net (208.106.96.193) 9.510 ms > 9.871 ms 9.194 ms > 4 200.ge-0-1-0.gw.equinix-sj.sonic.net (64.142.0.210) 11.965 ms > 11.870 ms 11.839 ms > 5 0.as0.gw2.equinix-sj.sonic.net (64.142.0.150) 11.928 ms 12.519 > ms 12.394 ms > 6 GigabitEthernet3-1.GW2.SJC7.ALTER.NET (157.130.194.17) 11.360 ms > 16.257 ms 11.268 ms > 7 0.so-0-0-1.XL4.SJC7.ALTER.NET (152.63.51.50) 11.729 ms 11.679 ms > 11.403 ms > 8 0.so-7-0-0.XL2.PAO1.ALTER.NET (152.63.113.21) 14.775 ms 17.455 > ms 0.so-5-0-0.XL2.PAO1.ALTER.NET (152.63.48.9) 15.548 ms > 9 POS7-0.GW6.PAO1.ALTER.NET (152.63.55.14) 12.886 ms 13.143 ms 13.029 > ms > 10 65.203.37.46 (65.203.37.46) 13.517 ms 14.708 ms 16.566 ms > 11 * * * > 12 * * * > ^C > > To find out more about router #3's egress port, I look at the IP address > for > router #4 (64.142.0.210), figure out what would be the other IP address in > the same /30 (64.142.0.209; hint: the lower address in a /30 pair always > ends in an odd number, and the higher address always ends in an even > number, > so if the address you know ends in an odd number, the other address in the > same /30 is going to be the next *higher* number, and if the address you > know is even, the other is going to be the next *lower* number), and do a > DNS reverse lookup of that address: > > *brent%* dig -x 64.142.0.209 > > ; <<>> DiG 9.4.3-P3 <<>> -x 64.142.0.209 > ;; global options: printcmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49382 > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 > > ;; QUESTION SECTION: > ;209.0.142.64.in-addr.arpa. IN PTR > > ;; ANSWER SECTION: > 209.0.142.64.in-addr.arpa. 259200 IN PTR > 200.ge-6-3-0.gw3.200p-sf.sonic.net. > > ;; Query time: 31 msec > ;; SERVER: 208.67.222.222#53(208.67.222.222) > ;; WHEN: Fri Nov 13 09:42:05 2009 > ;; MSG SIZE rcvd: 91 > > > Another handy tip from the presentation is that, since light travels > through > fiber optic cable at about 200 km (or 125 miles, if you prefer) per > millisecond, each 1 ms of delay shown by traceroute (which, remember, > is*round > trip* delay) should represent about 100 km (62.5 mi) of distance if the > delay were due entirely to the distance travelled (i.e., no queuing or > processing delays). Using that fact, you can see that 40ms for a packet to > go from San Francisco to New York (about 2500 miles, or 4000km) would be > "normal", but 40ms for a packet to go from San Francisco to San Jose (about > 50 miles, or 80km) would indicate a problem; it should take the packet less > than 1ms to cover that distance and back, so something else (congestion or > processing delays, for example) must account for the other 39ms. > > There's a *lot* more in this presentation, about more complex issues such > as > > - how the way in which routers handle traceroute packets can produce > biased results (most routers handle traceroute packets much more slowly > than > they handle "real" data packets, which can make things look much worse > than > they are) > - how asymmetric paths can lead you astray (traceroute only shows you the > path *to* a system, but if you're pulling lots of bytes from the system, > as would typically be the case with a remote server, you probably care > more > about the path back *from* the system, which might be totally different > - how using MPLS, which is increasingly common in carrier networks, can > lead to very confusing round-trip times in traceroute > > Anyway, if you ever use traceroute, I highly recommend that you review this > excellent presentation. I think you'll be pleasantly surprised at how much > you learn. > > Thanks to Strata Chalup of > Virtual.net<http://www.linkedin.com/in/virtualnet> for > bringing this very informative presentation to my attention. > -Brent > -- > Brent Chapman <> > Netomata, Inc. -- www.netomata.com > Making networks more cost-effective, reliable, and flexible by automating > network configuration > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~
