Re: A BGP issue?
On Tue, 2011-03-08 at 09:25 +0200, Hank Nussbacher wrote: At 21:49 07/03/2011 -0500, Patrick W. Gilmore wrote: On Mar 7, 2011, at 14:27, Greg Ihnen os10ru...@gmail.com wrote: I run a small network on a mission base in the Amazon jungle which is fed by a satellite internet connection. We had an outage from Feb 25th to the 28th where we had no connectivity with email, http/s, ftp, Skype would indicate it's connected but even chatting failed, basically everything stopped working except for ICMP. I could ping everywhere just fine. I started doing traceroutes and they all were very odd, all not reaching their destination and some hopping all over creation before dying. But if I did traceroute with ICMP it worked fine. Does this indicate our upstream (Bantel.net) had a BGP issue? Bantel blamed Hughesnet which is the service they resell. I'm wondering what kind of problem would let ping work fine but not any of the other protocols. It also seems odd that I could traceroute via UDP part way to a destination but then it would fail if the problem was my own provider. Thanks. If this is the wrong forum for this post I'm sorry and please just hit delete. If this is the wrong forum but you'd be kind enough to share your expertise please reply off-list. Thanks! Honestly, I would rate this as one of the most on-topic posts in a while. +1. When you have http working I suggest running: http://netalyzr.icsi.berkeley.edu/index.html to give you a benchmark of what your connection can do in the way of protocols. Regards, Hank Greg - you may want try doing pings with large packets. You may have MTU mismatch or some other problem with a link with lets small ICMP pings through but mangles or discards large packets. --vadim
Re: A BGP issue?
I have seen this kind of problems in our customers networks and this is motly related to MTU/MSS. Network reachability is fine but applciations does not work. Apply the MSS on the LAN interface which is around 100 bytes less than the MTU on the WAN interface. -Thanks, Viral. On 8 March 2011 16:11, Vadim Antonov a...@kotovnik.com wrote: On Tue, 2011-03-08 at 09:25 +0200, Hank Nussbacher wrote: At 21:49 07/03/2011 -0500, Patrick W. Gilmore wrote: On Mar 7, 2011, at 14:27, Greg Ihnen os10ru...@gmail.com wrote: I run a small network on a mission base in the Amazon jungle which is fed by a satellite internet connection. We had an outage from Feb 25th to the 28th where we had no connectivity with email, http/s, ftp, Skype would indicate it's connected but even chatting failed, basically everything stopped working except for ICMP. I could ping everywhere just fine. I started doing traceroutes and they all were very odd, all not reaching their destination and some hopping all over creation before dying. But if I did traceroute with ICMP it worked fine. Does this indicate our upstream (Bantel.net) had a BGP issue? Bantel blamed Hughesnet which is the service they resell. I'm wondering what kind of problem would let ping work fine but not any of the other protocols. It also seems odd that I could traceroute via UDP part way to a destination but then it would fail if the problem was my own provider. Thanks. If this is the wrong forum for this post I'm sorry and please just hit delete. If this is the wrong forum but you'd be kind enough to share your expertise please reply off-list. Thanks! Honestly, I would rate this as one of the most on-topic posts in a while. +1. When you have http working I suggest running: http://netalyzr.icsi.berkeley.edu/index.html to give you a benchmark of what your connection can do in the way of protocols. Regards, Hank Greg - you may want try doing pings with large packets. You may have MTU mismatch or some other problem with a link with lets small ICMP pings through but mangles or discards large packets. --vadim
Re: A BGP issue?
On Mar 7, 2011, at 10:19 PM, Patrick W. Gilmore wrote: On Mar 7, 2011, at 14:27, Greg Ihnen os10ru...@gmail.com wrote: I run a small network on a mission base in the Amazon jungle which is fed by a satellite internet connection. We had an outage from Feb 25th to the 28th where we had no connectivity with email, http/s, ftp, Skype would indicate it's connected but even chatting failed, basically everything stopped working except for ICMP. I could ping everywhere just fine. I started doing traceroutes and they all were very odd, all not reaching their destination and some hopping all over creation before dying. But if I did traceroute with ICMP it worked fine. Does this indicate our upstream (Bantel.net) had a BGP issue? Bantel blamed Hughesnet which is the service they resell. I'm wondering what kind of problem would let ping work fine but not any of the other protocols. It also seems odd that I could traceroute via UDP part way to a destination but then it would fail if the problem was my own provider. Thanks. If this is the wrong forum for this post I'm sorry and please just hit delete. If this is the wrong forum but you'd be kind enough to share your expertise please reply off-list. Thanks! Honestly, I would rate this as one of the most on-topic posts in a while. BGP only handles reachability, not higher level protocols. (Of course, you can h4x0r anything to do jus about anything, but we are talking the general case here.) If you can ping, BGP is working. If you can ping and cannot use TCP, then something other than BGP is at fault. I've seen strange things like someone enabling TCP compression (common on very small or very expensive links) one side but not the other, which then allowed ICMP and UDP but not TCP. It is a great way to annoy someone. See, I can ping, it must be your side! Have you tried TCP traceroute? Or telnetting to port 80? -- TTFN, patrick Patrick, Thank you very much! Thank you to everyone else who replied. I did try TCP traceroute and it failed too. I didn't have a machine to telnet to on port 80 but I did try an ssh tunnel on port and it failed too. From what everyone is saying it sounds like it was the satellite internet provider's compression scheme that was having trouble or some kind of an MTU issue. What I don't understand is why when using traceroute UDP/TCP/GRE I could get replies from some routers but not all routers to the destination, and why some routes were bizarre. If it was a failure of the sat internet provider's compression scheme or an MTU issue wouldn't traceroute UDP/TCP/GRE fail completely? What could have happened to my packets that would make them go only part way or go the wrong way? According to our satellite internet service provider Bantel the outage was system wide. Thank again! Greg
Re: A BGP issue?
On Mar 8, 2011, at 8:52 AM, Greg Ihnen wrote: On Mar 7, 2011, at 10:19 PM, Patrick W. Gilmore wrote: On Mar 7, 2011, at 14:27, Greg Ihnen os10ru...@gmail.com wrote: I run a small network on a mission base in the Amazon jungle which is fed by a satellite internet connection. We had an outage from Feb 25th to the 28th where we had no connectivity with email, http/s, ftp, Skype would indicate it's connected but even chatting failed, basically everything stopped working except for ICMP. I could ping everywhere just fine. I started doing traceroutes and they all were very odd, all not reaching their destination and some hopping all over creation before dying. But if I did traceroute with ICMP it worked fine. Does this indicate our upstream (Bantel.net) had a BGP issue? Bantel blamed Hughesnet which is the service they resell. I'm wondering what kind of problem would let ping work fine but not any of the other protocols. It also seems odd that I could traceroute via UDP part way to a destination but then it would fail if the problem was my own provider. Thanks. If this is the wrong forum for this post I'm sorry and please just hit delete. If this is the wrong forum but you'd be kind enough to share your expertise please reply off-list. Thanks! Honestly, I would rate this as one of the most on-topic posts in a while. BGP only handles reachability, not higher level protocols. (Of course, you can h4x0r anything to do jus about anything, but we are talking the general case here.) If you can ping, BGP is working. If you can ping and cannot use TCP, then something other than BGP is at fault. I've seen strange things like someone enabling TCP compression (common on very small or very expensive links) one side but not the other, which then allowed ICMP and UDP but not TCP. It is a great way to annoy someone. See, I can ping, it must be your side! Have you tried TCP traceroute? Or telnetting to port 80? I did try TCP traceroute and it failed too. I didn't have a machine to telnet to on port 80 but I did try an ssh tunnel on port and it failed too. Sure you do. Any web server will allow you to telnet to port 80. TiggerBook-Air3:~ patrick$ telnet www.yahoo.com 80 Trying 67.195.160.76... Connected to any-fp.wa1.b.yahoo.com. Escape character is '^]'. GET GET HEADTITLENot Found/TITLE/HEAD BODY BGCOLOR=white FGCOLOR=black FONT FACE=Helvetica,ArialB Your requested URL was not found./B/FONT !-- default Not Found response (404) -- /BODY Connection closed by foreign host. [In case it wasn't clear, I typed GET GET myself, just to have the web server respond with something.] From what everyone is saying it sounds like it was the satellite internet provider's compression scheme that was having trouble or some kind of an MTU issue. What I don't understand is why when using traceroute UDP/TCP/GRE I could get replies from some routers but not all routers to the destination, and why some routes were bizarre. If it was a failure of the sat internet provider's compression scheme or an MTU issue wouldn't traceroute UDP/TCP/GRE fail completely? What could have happened to my packets that would make them go only part way or go the wrong way? It was likely not MTU if you can traceroute to some places, but not others. Traceroute doesn't send or receive big packets. And I didn't really see anything terribly unusual in the traces you sent, other than some not completing. If you are talking about the Cogent one, with many routers per hop, that's just standard load balancing. -- TTFN, patrick
Re: A BGP issue?
On Mar 7, 2011, at 14:27, Greg Ihnen os10ru...@gmail.com wrote: I run a small network on a mission base in the Amazon jungle which is fed by a satellite internet connection. We had an outage from Feb 25th to the 28th where we had no connectivity with email, http/s, ftp, Skype would indicate it's connected but even chatting failed, basically everything stopped working except for ICMP. I could ping everywhere just fine. I started doing traceroutes and they all were very odd, all not reaching their destination and some hopping all over creation before dying. But if I did traceroute with ICMP it worked fine. Does this indicate our upstream (Bantel.net) had a BGP issue? Bantel blamed Hughesnet which is the service they resell. I'm wondering what kind of problem would let ping work fine but not any of the other protocols. It also seems odd that I could traceroute via UDP part way to a destination but then it would fail if the problem was my own provider. Thanks. If this is the wrong forum for this post I'm sorry and please just hit delete. If this is the wrong forum but you'd be kind enough to share your expertise please reply off-list. Thanks! Honestly, I would rate this as one of the most on-topic posts in a while. BGP only handles reachability, not higher level protocols. (Of course, you can h4x0r anything to do jus about anything, but we are talking the general case here.) If you can ping, BGP is working. If you can ping and cannot use TCP, then something other than BGP is at fault. I've seen strange things like someone enabling TCP compression (common on very small or very expensive links) one side but not the other, which then allowed ICMP and UDP but not TCP. It is a great way to annoy someone. See, I can ping, it must be your side! Have you tried TCP traceroute? Or telnetting to port 80? -- TTFN, patrick Here's some examples of the traceroutes I saved during the outage. Using UDP: Gregs-MacBook-Pro:~ GregIhnen$ traceroute metaconi.com traceroute to metaconi.com (70.32.39.205), 64 hops max, 52 byte packets 1 192.168.7.1 (192.168.7.1) 1541.165 ms 25.665 ms 39.211 ms 2 * * * 3 192.168.14.254 (192.168.14.254) 625.710 ms 860.264 ms 694.238 ms 4 192.168.180.5 (192.168.180.5) 645.666 ms 757.161 ms 664.785 ms 5 10.254.253.158 (10.254.253.158) 738.661 ms 801.487 ms 728.139 ms 6 fe11-0-5.miami1.mia.seabone.net (195.22.199.77) 726.884 ms 733.989 ms 647.736 ms 7 te3-4.miami7.mia.seabone.net (195.22.199.97) 740.233 ms 694.619 ms 685.464 ms 8 206.111.1.161.ptr.us.xo.net (206.111.1.161) 639.077 ms 741.495 ms 679.880 ms 9 te-4-1-0.rar3.miami-fl.us.xo.net (207.88.12.161) 650.312 ms 612.386 ms 660.452 ms 10 te-3-2-0.rar3.atlanta-ga.us.xo.net (207.88.12.5) 787.079 ms 725.495 ms 685.068 ms 11 te-11-0-0.rar3.washington-dc.us.xo.net (207.88.12.10) 760.002 ms 828.076 ms 702.041 ms 12 ae0d0.mcr2.chicago-il.us.xo.net (216.156.0.166) 719.324 ms 641.274 ms 689.997 ms 13 ae1d0.mcr1.chicago-il.us.xo.net (216.156.1.81) 669.613 ms 813.794 ms 737.211 ms 14 edge1.chi1.ubiquityservers.com (216.55.8.30) 729.875 ms 751.481 ms 730.088 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * Now here it is again doing traceroute via ICMP: Gregs-MacBook-Pro:~ GregIhnen$ traceroute -I metaconi.com traceroute to metaconi.com (70.32.39.205), 64 hops max, 72 byte packets 1 192.168.7.1 (192.168.7.1) 5.254 ms 3.059 ms 2.578 ms 2 * * * 3 192.168.14.254 (192.168.14.254) 1511.146 ms 711.304 ms 822.967 ms 4 192.168.180.5 (192.168.180.5) 712.672 ms 821.990 ms 713.009 ms 5 10.254.253.158 (10.254.253.158) 823.244 ms 711.764 ms 823.219 ms 6 fe11-0-5.miami1.mia.seabone.net (195.22.199.77) 712.640 ms 613.306 ms 614.429 ms 7 te3-4.miami7.mia.seabone.net (195.22.199.97) 823.232 ms 711.881 ms 823.166 ms 8 206.111.1.161.ptr.us.xo.net (206.111.1.161) 712.765 ms 822.398 ms 712.531 ms 9 te-4-1-0.rar3.miami-fl.us.xo.net (207.88.12.161) 822.809 ms 920.831 ms 712.399 ms 10 te-3-2-0.rar3.atlanta-ga.us.xo.net (207.88.12.5) 823.288 ms 711.478 ms 822.887 ms 11 te-11-0-0.rar3.washington-dc.us.xo.net (207.88.12.10) 712.705 ms 822.287 ms 712.713 ms 12 * ae0d0.mcr2.chicago-il.us.xo.net (216.156.0.166) 738.656 ms 919.752 ms 13 ae1d0.mcr1.chicago-il.us.xo.net (216.156.1.81) 921.381 ms 920.884 ms 1228.683 ms 14 edge1.chi1.ubiquityservers.com (216.55.8.30) 921.560 ms 920.482 ms 921.634 ms 15 relativity.mrk.com (70.32.39.205) 880.318 ms 753.150 ms 823.285 ms Gregs-MacBook-Pro:~ GregIhnen$ Here's an example of a UDP traceroute going all over creation: Gregs-MacBook-Pro:~ GregIhnen$ traceroute skype.com traceroute to skype.com (78.141.177.7), 64 hops max, 52 byte packets 1 192.168.7.1 (192.168.7.1) 18.939 ms 4.596 ms 27.124 ms 2 * * * 3
Re: A BGP issue?
At 21:49 07/03/2011 -0500, Patrick W. Gilmore wrote: On Mar 7, 2011, at 14:27, Greg Ihnen os10ru...@gmail.com wrote: I run a small network on a mission base in the Amazon jungle which is fed by a satellite internet connection. We had an outage from Feb 25th to the 28th where we had no connectivity with email, http/s, ftp, Skype would indicate it's connected but even chatting failed, basically everything stopped working except for ICMP. I could ping everywhere just fine. I started doing traceroutes and they all were very odd, all not reaching their destination and some hopping all over creation before dying. But if I did traceroute with ICMP it worked fine. Does this indicate our upstream (Bantel.net) had a BGP issue? Bantel blamed Hughesnet which is the service they resell. I'm wondering what kind of problem would let ping work fine but not any of the other protocols. It also seems odd that I could traceroute via UDP part way to a destination but then it would fail if the problem was my own provider. Thanks. If this is the wrong forum for this post I'm sorry and please just hit delete. If this is the wrong forum but you'd be kind enough to share your expertise please reply off-list. Thanks! Honestly, I would rate this as one of the most on-topic posts in a while. +1. When you have http working I suggest running: http://netalyzr.icsi.berkeley.edu/index.html to give you a benchmark of what your connection can do in the way of protocols. Regards, Hank
RE: Comcast BGP issue?
Sorry, my typo. The Net is 75.150.64.0/18 -Original Message- From: Wallace Keith Sent: Thursday, February 10, 2011 12:20 PM To: nanog@nanog.org Subject: Comcast BGP issue? Not quite sure what the issue is, but I suspect Comcast announcements are not quite right? Trying to get from Verizon Business to a Comcast address in NH (on 75.15.64.0/18), and it's going through San Jose. Anyone else having similar issues or suggestions? Opened a ticket with Comcast, but they want to check the cable modem (sigh) 6 6 ms 6 ms 6 ms 0.so-3-2-0.XL3.BOS4.ALTER.NET [152.63.22.178] 713 ms14 ms13 ms 0.xe-6-1-2.XL3.NYC4.ALTER.NET [152.63.0.166] 814 ms46 ms13 ms 0.ae3.BR3.NYC4.ALTER.NET [152.63.16.181] 914 ms14 ms13 ms te-7-0-0.edge2.NewYork2.level3.net [4.68.110.69] 1013 ms13 ms14 ms vlan52.ebr2.NewYork2.Level3.net [4.69.138.254] 1114 ms14 ms14 ms ae-6-6.ebr2.NewYork1.Level3.net [4.69.141.21] 1285 ms88 ms90 ms ae-2-2.ebr4.SanJose1.Level3.net [4.69.135.185] 1385 ms85 ms89 ms ae-94-94.csw4.SanJose1.Level3.net [4.69.134.254] 1483 ms83 ms84 ms ae-4-99.edge1.SanJose1.Level3.net [4.68.18.206] 1597 ms95 ms95 ms COMCAST-IP.edge1.SanJose1.Level3.net [4.79.43.134] 16 125 ms 125 ms 124 ms pos-0-14-0-0-cr01.denver.co.ibone.comcast.net [68.86.85.129] 17 135 ms 129 ms 125 ms pos-0-14-0-0-cr01.chicago.il.ibone.comcast.net [68.86.85.117] 18 122 ms 121 ms 121 ms pos-1-15-0-0-ar01.woburn.ma.boston.comcast.net [68.86.93.122] 19 128 ms 127 ms 127 ms po-66-ur02.manchester.nh.boston.comcast.net [68.85.69.170] 20 123 ms 121 ms 121 ms 68.85.161.226 21 116 ms 118 ms 119 ms x-x-x-x-newengland.hfc.comcastbusiness.net [75.150.80.x] -Keith
