UPDATED - Comcast enables 6to4 relays
Enabled two more 6to4 relays this morning. :) John On 8/31/10 1:14 AM, "Mikael Abrahamsson" wrote: > On Mon, 30 Aug 2010, Jack Bates wrote: > >> I'm sure, like us, you looked at what was involved and said, "eh, easier >> to just provide native v6 than deal with that mess." 6to4 is definitely >> a more friendly protocol for the network engineer. > > End users are using 6to4 and Teredo, if an ISP isn't providing their own > relays, someone else is and the performance might be good or bad. > > Same logic applies to Teredo as to 6to4 and why if you're an ISP who > cares, you should run your own. Your customers are using both, whether > they know it or not. = John Jason Brzozowski Comcast Cable e) mailto:john_brzozow...@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =
Re: Comcast enables 6to4 relays
On 2010-08-31 08:22, Mitchell Warden wrote: [..] > Is there a reason not to advertise more specific prefixes from 2002::/16 to > ensure that traffic for your v4 routes comes back to your own 6to4 router? > > If for example all my users have v4 addresses in 192.0.2.0/24, I could > advertise 2002:C002:::/40 instead of or in addition to the full 2002::/16. The RFC forbids that with a good reason, as then we'll end up importing the IPv4 BGP table into IPv6... not something we want to see (unless one loves to import 300k routes in there, I guess people will really start whining about that though ;). Greets, Jeroen
Re: Comcast enables 6to4 relays
> The list seems to be showing relays that announce both the IPv4 and the > IPv6 anycast prefixes. > > I have noticed a number of deployments that announce the (in)famous IPv4 > prefix and then consider their deployment complete. I suspect that there > is a lack of 2002::/16 announcements and this would be contributing to > the regular problems with return paths. > > Obviously the IPv6 content networks benefit the most from having a relay > translating back to IPv4. > > Anyone have experience with this? > > -- > Graham Beneke > > Is there a reason not to advertise more specific prefixes from 2002::/16 to ensure that traffic for your v4 routes comes back to your own 6to4 router? If for example all my users have v4 addresses in 192.0.2.0/24, I could advertise 2002:C002:::/40 instead of or in addition to the full 2002::/16. Cheers. Mitchell
Re: Comcast enables 6to4 relays
On Mon, 30 Aug 2010, Jack Bates wrote: I'm sure, like us, you looked at what was involved and said, "eh, easier to just provide native v6 than deal with that mess." 6to4 is definitely a more friendly protocol for the network engineer. End users are using 6to4 and Teredo, if an ISP isn't providing their own relays, someone else is and the performance might be good or bad. Same logic applies to Teredo as to 6to4 and why if you're an ISP who cares, you should run your own. Your customers are using both, whether they know it or not. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Comcast enables 6to4 relays
On 30/08/2010 23:47, Franck Martin wrote: found it: http://www.bgpmon.net/6to4.php?week=4 Not what I call a big list, considering... The list seems to be showing relays that announce both the IPv4 and the IPv6 anycast prefixes. I have noticed a number of deployments that announce the (in)famous IPv4 prefix and then consider their deployment complete. I suspect that there is a lack of 2002::/16 announcements and this would be contributing to the regular problems with return paths. Obviously the IPv6 content networks benefit the most from having a relay translating back to IPv4. Anyone have experience with this? -- Graham Beneke
Re: Comcast enables 6to4 relays
On Mon, Aug 30, 2010 at 3:34 PM, Jack Bates wrote: > John Jason Brzozowski wrote: >> >> Hey Bill, >> >> No plans for Teredo at this time. >> > > I'm sure, like us, you looked at what was involved and said, "eh, easier to > just provide native v6 than deal with that mess." 6to4 is definitely a more > friendly protocol for the network engineer. 100% agree about deploying native IPv6 being better than any "transition mechanism". Time spent deploying IPv6 native is time well spent with real long term payback (ROI ...). Any significant amount of time spent on any transition mechanism that is not strategic to your business, especially unmanaged off-network tunnels, will be painful. Better to work on the piloting a long term solution that will scale and fits the long term reality of living in a IPv4-only + IPv6-only + dual-stack internet. Although i think 6to4 has hurt IPv6 adoption more than help it, i believe comcast is doing the right thing given the fact that many people are running 6to4 and don't even know it and without on-net relays, the experience leaves a lot to be desired.. Regards, Cameron http://groups.google.com/group/tmoipv6beta
Re: Comcast enables 6to4 relays
Others may correct me, but... Franck Martin wrote: 5 2002:7114:4a9d::1 274.299 ms [mtu: 1480] 6 2002:7114:4a9d:0: 299.939 ms [*mtu: 1422] So I suspect on return path I use a HE.Net relay? Yes, and it appears that your host is replying back to the office. And yes I agree it is such a PITA to troubleshoot, I cannot pinpoint the issue. Nor I cannot see if I'm the only one with this kind of troubles, I suspect most just give up immediately... Given that you seem to complete the trace one direction, but not the other, I can think of 2 things, 1) Your originating host may be breaking PMTU (so the packet you send is too large and doesn't make it, you never resend a smaller packet, but it works when tracerouting from the other side due to PMTU working in that direction and you are responding with the same size packet). 2) If hosts in the middle on your traceroute to the destination don't have access to 6to4 relay, they won't be able to reply back to you, so you may end up with timeout hops before getting to the endpoint. Jack
Re: Comcast enables 6to4 relays
John Jason Brzozowski wrote: Hey Bill, No plans for Teredo at this time. I'm sure, like us, you looked at what was involved and said, "eh, easier to just provide native v6 than deal with that mess." 6to4 is definitely a more friendly protocol for the network engineer. Jack
Re: Comcast enables 6to4 relays
if only I had a static IPv4 address, but who gives such luxury for free nowadays to a end user? I'm in an end user configuration here The office configuration is working perfectly fine. I'm talking about my experience as a "end user" just enabling IPv6 on my airport extreme and trying to figure out what is happening On my return path I got: traceroute from 2001:470:1:74::3 to 2002:7114:4a9d:0: 1 2001:470:1:74::1 5.582 ms [mtu: 1500] 2 2001:470:0:2d::2 0.483 ms [mtu: 1500] 3 2001:470:0:30::2 173.747 ms [mtu: 1500] 4 2001:470:0:13b::2 0.848 ms [mtu: 1500] 5 2002:7114:4a9d::1 274.299 ms [mtu: 1480] 6 2002:7114:4a9d:0: 299.939 ms [*mtu: 1422] So I suspect on return path I use a HE.Net relay? And yes I agree it is such a PITA to troubleshoot, I cannot pinpoint the issue. Nor I cannot see if I'm the only one with this kind of troubles, I suspect most just give up immediately... - Original Message - From: "Jack Bates" To: "Franck Martin" Cc: "John Jason Brzozowski" , "NANOG" Sent: Tuesday, 31 August, 2010 10:14:39 AM Subject: Re: Comcast enables 6to4 relays Franck Martin wrote: > Well I found my 6to4 gateway: > and I have so much issues with 6to4 that I have decided to disable it at home > (airport extreme). I found out PTB was not transmitted and using scamper and > the help of Matthew Luckie there is an odd MTU of 1422 from Internet to me. I > suspect the 6to4 relay did not put the MTU to 1280 to be on the safe side... > (I saw a recommendation like that on the net). > Actually, their MTU won't matter. 6to4 is EXTREMELY asymmetric when using relays. The return traffic to you will be via the first router than can support 6to4, not the relay you sent packets to. This is why troubleshooting 6to4 is such a PITA. If your airport supports static tunnels, set one up at tunnel brokers and be done with it. Jack
Re: Comcast enables 6to4 relays
Franck Martin wrote: Well I found my 6to4 gateway: and I have so much issues with 6to4 that I have decided to disable it at home (airport extreme). I found out PTB was not transmitted and using scamper and the help of Matthew Luckie there is an odd MTU of 1422 from Internet to me. I suspect the 6to4 relay did not put the MTU to 1280 to be on the safe side... (I saw a recommendation like that on the net). Actually, their MTU won't matter. 6to4 is EXTREMELY asymmetric when using relays. The return traffic to you will be via the first router than can support 6to4, not the relay you sent packets to. This is why troubleshooting 6to4 is such a PITA. If your airport supports static tunnels, set one up at tunnel brokers and be done with it. Jack
Re: Comcast enables 6to4 relays
Hey Bill, No plans for Teredo at this time. John On 8/30/10 5:57 PM, "Bill Fehring" wrote: > On Sat, Aug 28, 2010 at 10:49, John Jason Brzozowski > wrote: > >> >> As we started our IPv6 trials, we began to observe an increase in 6to4 relay >> traffic. 6to4 is a transition mechanism built into some operating systems >> and home gateways. While it is not a transition technology that Comcast >> planned to invest in due to limitations related to performance, we did >> observe poor performance when 6to4 was used by our customers. In many cases, >> these customers were not even aware that 6to4 was enabled by default or that >> their device or operating system was attempting to use 6to4 to communicate >> with IPv6 resources on the Internet. >> >> In most cases, we observed that 6to4-enabled operating systems and devices >> were attempting to use a 6to4 relay infrastructure hosted by a midwestern >> university. In order to improve the Internet experience for Comcast >> customers who are using 6to4, whether knowingly or not, we have decided to >> operate 6to4 relays on a temporary, trial basis. >> >> Comcast has decided to deploy 6to4 relays in five locations around our >> network to improve performance and predictability, as compared to operating >> relays from a single location. These 6to4 relays are available via the >> standard 6to4 Anycast IP address, according to RFC 3068, which is >> 192.88.99.1. Devices attempting to use 6to4 within our network should >> automatically discover and utilize these new 6to4 relays, without end user >> intervention or configuration. >> >> The first pair of these relays was activated today. We plan to activate the >> remaining three within the next seven to ten days. We plan to monitor the >> performance of the 6to4 relays, to measure any beneficial effects resulting >> from adding these elements to our network. As our IPv6 trials evolve and we >> develop our plans for 2011 and beyond, we will assess our plans to support >> 6to4 moving forward. >> > > > Hi John, > > First of all, that's great news -- I think it will help a lot. > > Have you also considered deploying Teredo relays? I'm guessing that > there are quite a few Windows Vista+ systems that could benefit from > having a few closer Teredo relays and it's probably a similar amount > of traffic that you're seeing compared to 6to4 tunnels. > > Best, > > Bill Fehring = John Jason Brzozowski Comcast Cable e) mailto:john_brzozow...@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =
Re: Comcast enables 6to4 relays
I actually agree with the below. Using whatever you learn "today" via BGP does not appear to be a good plan. 6to4 in particular becomes very unpredictable and does in fact contribute to brokenness. I am not saying deploying your own will make 6to4 good or great, it will however, help to make it less broken and more controllable. If operators deployed their own it would likely be beneficial to their subscribers, particularly if there is a lot of 6to4. I would also go out on a limb and say that absent and poorly deployed 6to4 relays collectively play hugely into the perception of brokenness. There was a noticeable difference deploying our own 6to4 relays compared to using the next or first available. I am not saying the other relay was poorly run, just saying it is different having one on-net versus off-net. John On 8/30/10 5:44 PM, "Jack Bates" wrote: > > Franck Martin wrote: >> Is there a list of 6to4 relays? >> >> I'm curious. >> >> Also, I'm also curious to know if ISPs in Europe (which are more advanced in >> IPv6 deployment) have experienced the same issues? >> >> > > Sprint has one which is absolutely horrible (or was a year or two ago). > I'd recommend any and every ISP to setup a 6to4, even if it runs over a > v6 tunnel to HE. Accidentally getting one from someone else can give you > exceptionally broken 6to4 connectivity. Being anycast, I'd say > routeviews might be a good place to check for some, but often times they > are hidden within the networks they serve. > > That being said, 6to4 itself is often horrible. It works fine if you are > talking 6to4 direct to the remote site (vs using a relay), but relays > often break and are hard to troubleshoot due to their nature. > > In the last year, my 6to4 tunnel has peaked at 6.44mb/s (1day average) > but more common peaks are 150-250kb/s (5min average). > > My tunnel to he.net was running 4-8 times that including the 6to4 relay > serving all my customers + native traffic for 15 hosts and 2 servers. > It's hard to get accurate on recent traffic loads as much of my v6 > traffic shifted to dual stacked peers and I don't have a method of > separating the v4/v6 traffic in the graphs (me thinks it's time to test > ipv6 over mpls). > > > Jack = John Jason Brzozowski Comcast Cable e) mailto:john_brzozow...@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =
Re: Comcast enables 6to4 relays
Yes this is the list of visible relays as seen from the BGP backbone monitoring... If you don't offer your relays to the rest of the world, they won't show up there... - Original Message - From: "John Jason Brzozowski" To: "Franck Martin" Cc: "NANOG" Sent: Tuesday, 31 August, 2010 10:09:17 AM Subject: Re: Comcast enables 6to4 relays The Comcast 6to4 relays are not on this list, perhaps this is a list of open ones? John On 8/30/10 5:47 PM, "Franck Martin" wrote: > found it: > > http://www.bgpmon.net/6to4.php?week=4 > > Not what I call a big list, considering... > > - Original Message - > From: "Franck Martin" > To: "John Jason Brzozowski" > Cc: "NANOG" > Sent: Tuesday, 31 August, 2010 9:21:58 AM > Subject: Re: Comcast enables 6to4 relays > > Is there a list of 6to4 relays? > > I'm curious. > > Also, I'm also curious to know if ISPs in Europe (which are more advanced in > IPv6 deployment) have experienced the same issues? > > > > = John Jason Brzozowski Comcast Cable e) mailto:john_brzozow...@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =
Re: Comcast enables 6to4 relays
Well I found my 6to4 gateway: traceroute to 192.88.99.1 (192.88.99.1), 30 hops max, 40 byte packets 10 te3-3.ccr01.ind01.atlas.cogentco.com (154.54.3.30) [AS174] 244.965 ms 244.964 ms 244.952 ms 11 38.20.52.226 (38.20.52.226) [AS174] 244.336 ms 38.20.52.222 (38.20.52.222) [AS174] 244.300 ms 266.250 ms 12 38.104.214.6 (38.104.214.6) [AS174] 326.141 ms 326.139 ms 376.926 ms 13 ge-7-0-0.103.rtr.ictc.indiana.gigapop.net (149.165.254.142) [AS10680] 612.357 ms 612.367 ms 612.358 ms 14 rtr3.ul.indiana.gigapop.net (149.165.255.129) [AS10680] 376.777 ms 319.641 ms * and I have so much issues with 6to4 that I have decided to disable it at home (airport extreme). I found out PTB was not transmitted and using scamper and the help of Matthew Luckie there is an odd MTU of 1422 from Internet to me. I suspect the 6to4 relay did not put the MTU to 1280 to be on the safe side... (I saw a recommendation like that on the net). I have seen that the new hardware of the airport extreme has a new firmware that does more IPv6 magic, but old hardware are not yet benefiting this new firmware... Once a new firmware comes I'll re-enable and see... PS: I found scamper to be a very good troubleshooting tool (once you know what to do) and wish it would be on all OS, like traceroute and now tracepath is... - Original Message - From: "Jack Bates" To: "Franck Martin" Cc: "John Jason Brzozowski" , "NANOG" Sent: Tuesday, 31 August, 2010 9:44:11 AM Subject: Re: Comcast enables 6to4 relays Franck Martin wrote: > Is there a list of 6to4 relays? > > I'm curious. > > Also, I'm also curious to know if ISPs in Europe (which are more advanced in > IPv6 deployment) have experienced the same issues? > > Sprint has one which is absolutely horrible (or was a year or two ago). I'd recommend any and every ISP to setup a 6to4, even if it runs over a v6 tunnel to HE. Accidentally getting one from someone else can give you exceptionally broken 6to4 connectivity. Being anycast, I'd say routeviews might be a good place to check for some, but often times they are hidden within the networks they serve. That being said, 6to4 itself is often horrible. It works fine if you are talking 6to4 direct to the remote site (vs using a relay), but relays often break and are hard to troubleshoot due to their nature.
Re: Comcast enables 6to4 relays
The Comcast 6to4 relays are not on this list, perhaps this is a list of open ones? John On 8/30/10 5:47 PM, "Franck Martin" wrote: > found it: > > http://www.bgpmon.net/6to4.php?week=4 > > Not what I call a big list, considering... > > - Original Message - > From: "Franck Martin" > To: "John Jason Brzozowski" > Cc: "NANOG" > Sent: Tuesday, 31 August, 2010 9:21:58 AM > Subject: Re: Comcast enables 6to4 relays > > Is there a list of 6to4 relays? > > I'm curious. > > Also, I'm also curious to know if ISPs in Europe (which are more advanced in > IPv6 deployment) have experienced the same issues? > > > > = John Jason Brzozowski Comcast Cable e) mailto:john_brzozow...@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =
Re: Comcast enables 6to4 relays
In a message written on Tue, Aug 31, 2010 at 09:47:14AM +1200, Franck Martin wrote: > found it: > > http://www.bgpmon.net/6to4.php?week=4 > > Not what I call a big list, considering... Note that these are people willing to provide a 6to4 relay free to the entire Internet. There are plenty of people who offer 6to4 inside their own network for their own customers, but never advertise the prefix to world+dog. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ pgpBilP6sqwtS.pgp Description: PGP signature
Re: Comcast enables 6to4 relays
On Sat, Aug 28, 2010 at 10:49, John Jason Brzozowski wrote: > > As we started our IPv6 trials, we began to observe an increase in 6to4 relay > traffic. 6to4 is a transition mechanism built into some operating systems > and home gateways. While it is not a transition technology that Comcast > planned to invest in due to limitations related to performance, we did > observe poor performance when 6to4 was used by our customers. In many cases, > these customers were not even aware that 6to4 was enabled by default or that > their device or operating system was attempting to use 6to4 to communicate > with IPv6 resources on the Internet. > > In most cases, we observed that 6to4-enabled operating systems and devices > were attempting to use a 6to4 relay infrastructure hosted by a midwestern > university. In order to improve the Internet experience for Comcast > customers who are using 6to4, whether knowingly or not, we have decided to > operate 6to4 relays on a temporary, trial basis. > > Comcast has decided to deploy 6to4 relays in five locations around our > network to improve performance and predictability, as compared to operating > relays from a single location. These 6to4 relays are available via the > standard 6to4 Anycast IP address, according to RFC 3068, which is > 192.88.99.1. Devices attempting to use 6to4 within our network should > automatically discover and utilize these new 6to4 relays, without end user > intervention or configuration. > > The first pair of these relays was activated today. We plan to activate the > remaining three within the next seven to ten days. We plan to monitor the > performance of the 6to4 relays, to measure any beneficial effects resulting > from adding these elements to our network. As our IPv6 trials evolve and we > develop our plans for 2011 and beyond, we will assess our plans to support > 6to4 moving forward. > Hi John, First of all, that's great news -- I think it will help a lot. Have you also considered deploying Teredo relays? I'm guessing that there are quite a few Windows Vista+ systems that could benefit from having a few closer Teredo relays and it's probably a similar amount of traffic that you're seeing compared to 6to4 tunnels. Best, Bill Fehring
Re: Comcast enables 6to4 relays
Franck Martin wrote: Is there a list of 6to4 relays? I'm curious. Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues? Sprint has one which is absolutely horrible (or was a year or two ago). I'd recommend any and every ISP to setup a 6to4, even if it runs over a v6 tunnel to HE. Accidentally getting one from someone else can give you exceptionally broken 6to4 connectivity. Being anycast, I'd say routeviews might be a good place to check for some, but often times they are hidden within the networks they serve. That being said, 6to4 itself is often horrible. It works fine if you are talking 6to4 direct to the remote site (vs using a relay), but relays often break and are hard to troubleshoot due to their nature. In the last year, my 6to4 tunnel has peaked at 6.44mb/s (1day average) but more common peaks are 150-250kb/s (5min average). My tunnel to he.net was running 4-8 times that including the 6to4 relay serving all my customers + native traffic for 15 hosts and 2 servers. It's hard to get accurate on recent traffic loads as much of my v6 traffic shifted to dual stacked peers and I don't have a method of separating the v4/v6 traffic in the graphs (me thinks it's time to test ipv6 over mpls). Jack
Re: Comcast enables 6to4 relays
found it: http://www.bgpmon.net/6to4.php?week=4 Not what I call a big list, considering... - Original Message - From: "Franck Martin" To: "John Jason Brzozowski" Cc: "NANOG" Sent: Tuesday, 31 August, 2010 9:21:58 AM Subject: Re: Comcast enables 6to4 relays Is there a list of 6to4 relays? I'm curious. Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
Re: Comcast enables 6to4 relays
Is there a list of 6to4 relays? I'm curious. Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
Re: Did your BGP crash today?
On Mon, Aug 30, 2010 at 15:55, Jack Bates wrote: > ... > As good a place to break in on the thread as any, I guess. Randy and others > believe more testing should have been done. I'm not completely sure they > didn't test against XR. They very likely could have tested in a 1 on 1 > connection and everything looked fine. > > I don't know the full details, but at what point did the corruption appear, > and was it visible? We know that it was corrupt on the output which caused > peer resets, but was it necessarily visible in the router itself? > > Do we require a researcher to setup a chain of every vender BGP speaker in > every possible configuration and order to verify a bug doesn't cause things > to break? In this case, one very likely would need an XR receiving and > transmitting updates to detect the failure, so no less than 3 routers with > the XR in the middle. > > What about individual configurations? Perhaps the update is received and > altered by one vendor due to specific configurations, sent to the next > vendor, accepted and altered (due to the first alteration, where as it > wouldn't be altered if the original update had been received) which causes > the next vendor to reset. Then we add to this that it may pass silently > through several middle vendor routers without problems and we realize the > scope of such problems and why connecting to the Internet is so > unpredictable. I am not aware that anyone has provided the complete details at this point which would include any test plans that may have been performed. From what I have been able to discern, it does seem likely that a test plan that would have caught this almost had to know of the specific issue in advance. More testing would have been better, but there is just too much variability out there to assure you can do a complete test. I am also not aware that the introduction of the attribute was announced to the usual operational lists in advance "just in case" (Ok, in this case, I mean NANOG). This, is my mind, is actually the bigger faux pas. An "Oh S***" moment has happened to most of us. It probably will happen again to many of us. But letting people know in advance of scheduled changes is the important thing. I would hope that in the future researchers will commit to test plans to (at least) all the major vendor BGP speakers (which, I admit, would likely not have caught this issue), and that before introducing such "new" attributes into the "Internet", they would announce it to the usual operational lists, again, "just in case". But my hopes are often dashed. Gary
Re: Life experience: Cisco ASR9000 vs. Juniper MX960
We're planning to do exactly the same, but not in parallel. ASR9k first (in September), MX960 after. I'll be interested in your results (and everyone else), especially in the areas of L2VPN, M(V)PLS, OSPF and everything Carrier Ethernet related. -- Tassos Jason Lixfeld wrote on 30/08/2010 21:27: I'm looking for feedback on Cisco's ASR9000 vs. Juniper MX960. We're evaluating the two and are in the process of testing these two platforms and I'd like to try to have a leg up on testing if there are any gotchas that have crept up in the real world. What we're looking for information on are problems that have arisen in these areas: - Interface linerate issues (oversubscription, etc) - Backplane/slot capacity issues (oversubscription, etc) - Supervisor/RE switchover - DC PDUs and power redundancy - BGP4/MP-BGP - IPv6 - L2VPN - L3VPN - VPLS - LDP - RSVP - MPLS-TE - MPLS-FRR - OSPF - ISIS - In the case of Juniper, interoperability with Cisco for MPLS and associated protocols and features. - Anything else you'd like to mention, good or bad. If anyone is interested, I'll compile the results and provide them to whomever. Thanks in advance.
Life experience: Cisco ASR9000 vs. Juniper MX960
I'm looking for feedback on Cisco's ASR9000 vs. Juniper MX960. We're evaluating the two and are in the process of testing these two platforms and I'd like to try to have a leg up on testing if there are any gotchas that have crept up in the real world. What we're looking for information on are problems that have arisen in these areas: - Interface linerate issues (oversubscription, etc) - Backplane/slot capacity issues (oversubscription, etc) - Supervisor/RE switchover - DC PDUs and power redundancy - BGP4/MP-BGP - IPv6 - L2VPN - L3VPN - VPLS - LDP - RSVP - MPLS-TE - MPLS-FRR - OSPF - ISIS - In the case of Juniper, interoperability with Cisco for MPLS and associated protocols and features. - Anything else you'd like to mention, good or bad. If anyone is interested, I'll compile the results and provide them to whomever. Thanks in advance.
Re: Did your BGP crash today?
At 12:40 PM 8/30/2010, Kevin Oberman wrote: This only way they could have caught this one was to have tested to a CRS which had another router to which it was announcing the attribute in a mal-formed packet. Worse, the resets should just keep happening as the CRS would still have the route with the unknown attribute which would just generate another malformed update to cause the session to reset again. While it may be possible to recover from something like this, it sure would not be easy. We experienced something like this a year ago on a couple of quagga boxes. At least we had source code to go through and resources to make use of that source code to find the problem and implement a quick work around. Its for situations like this, debugging logging is ohhh so important. What did people do in this case to identify the issue ? Did you just pass it off to your vendor ? or did anyone try to diagnose it locally ? If so, what did you do ? ---Mike -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: ober...@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 Mike Tancsa, tel +1 519 651 3400 Sentex Communications,m...@sentex.net Providing Internet since 1994www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike
Re: Did your BGP crash today?
> Date: Mon, 30 Aug 2010 10:55:03 -0500 > From: Jack Bates > > > Florian Weimer wrote: > > This whole thread is quite schizophrenic because the consensus appears > > to be that (a) a *researcher is not to blame* for sending out a BGP > > message which eventually leads to session resets, and (b) an > > *implementor is to blame* for sending out a BGP messages which > > eventually leads to session resets. You really can't have it both > > ways. > > > > As good a place to break in on the thread as any, I guess. Randy and > others believe more testing should have been done. I'm not completely > sure they didn't test against XR. They very likely could have tested in > a 1 on 1 connection and everything looked fine. > > I don't know the full details, but at what point did the corruption > appear, and was it visible? We know that it was corrupt on the output > which caused peer resets, but was it necessarily visible in the router > itself? > > Do we require a researcher to setup a chain of every vender BGP speaker > in every possible configuration and order to verify a bug doesn't cause > things to break? In this case, one very likely would need an XR > receiving and transmitting updates to detect the failure, so no less > than 3 routers with the XR in the middle. > > What about individual configurations? Perhaps the update is received and > altered by one vendor due to specific configurations, sent to the next > vendor, accepted and altered (due to the first alteration, where as it > wouldn't be altered if the original update had been received) which > causes the next vendor to reset. Then we add to this that it may pass > silently through several middle vendor routers without problems and we > realize the scope of such problems and why connecting to the Internet is > so unpredictable. This only way they could have caught this one was to have tested to a CRS which had another router to which it was announcing the attribute in a mal-formed packet. Worse, the resets should just keep happening as the CRS would still have the route with the unknown attribute which would just generate another malformed update to cause the session to reset again. While it may be possible to recover from something like this, it sure would not be easy. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: ober...@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
Off-Topic: Seeking sponsor for infrastructure related site.
First of all i want to apologize for the off topic nature of this post. I am seeking sponsorship for one of my hobby project. My needs is one or two dedicated servers on a stable network in USA and or Europe. DNSDigger.com is one of my hobby projects that has outgrown my own means of hosting as i am on 75% sick leave from my ordinary job. DNSDigger is a database of domain and hostnames that is resolved into IP-adresses and can show what domains a IP-adress (or IP-span) has "behind". The data is collected by downloading com/net-zone from Verisign and http spidering. Then "simply" resolving all the hundreds of millions of hostnames and make it search able. This sounds like a bandwidth hog but i actually run all this from a home private cable connection (dont tell anyone ;) and i have had no troubles except for outgrowing my Pentium D "server". If you find this service fun and valuable and have the means of providing an high end dedicated server plus hosting i would love to hear from you. An "powered by "-link/button and information page will be offered. /John Stromstedt (j...@netrogenic.com)
Re: Did your BGP crash today?
Florian Weimer wrote: This whole thread is quite schizophrenic because the consensus appears to be that (a) a *researcher is not to blame* for sending out a BGP message which eventually leads to session resets, and (b) an *implementor is to blame* for sending out a BGP messages which eventually leads to session resets. You really can't have it both ways. As good a place to break in on the thread as any, I guess. Randy and others believe more testing should have been done. I'm not completely sure they didn't test against XR. They very likely could have tested in a 1 on 1 connection and everything looked fine. I don't know the full details, but at what point did the corruption appear, and was it visible? We know that it was corrupt on the output which caused peer resets, but was it necessarily visible in the router itself? Do we require a researcher to setup a chain of every vender BGP speaker in every possible configuration and order to verify a bug doesn't cause things to break? In this case, one very likely would need an XR receiving and transmitting updates to detect the failure, so no less than 3 routers with the XR in the middle. What about individual configurations? Perhaps the update is received and altered by one vendor due to specific configurations, sent to the next vendor, accepted and altered (due to the first alteration, where as it wouldn't be altered if the original update had been received) which causes the next vendor to reset. Then we add to this that it may pass silently through several middle vendor routers without problems and we realize the scope of such problems and why connecting to the Internet is so unpredictable. Jack
RE: Recycling old cabling?
Assumption: Construction guys are present. 1. Dump cable in large pile on the floor 2. Yell "Does anybody want this copper?" 3. Use broom to fend of multiple takers 4. Tell the guy who wants it that he can have it as long as he hauls it away Not that I've ever done this of course... The information in this email and any attachments are for the sole use of the intended recipient and may contain privileged and confidential information. If you are not the intended recipient, any use, disclosure, copying or distribution of this message or attachment is strictly prohibited. We have taken precautions to minimize the risk of transmitting software viruses, but we advise you to carry out your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. If you believe that you have received this email in error, please contact the sender immediately and delete the email and all of its attachments From: Patrik Wallstrom [mailto:pa...@blipp.com] Sent: Thu 8/19/2010 10:12 AM To: nanog@nanog.org Subject: Re: Recycling old cabling? On Aug 18, 2010, at 8:45 AM, Mikael Abrahamsson wrote: > On Wed, 18 Aug 2010, khatfi...@socllc.net wrote: > >> More companies recycle and properly dispose of equipment than they did ten >> years ago. Yet, if they aren't being looked at to be "green" or something >> along those lines then many choose the cheapest route (the dumpster). > > The amazing thing is sometimes they will pay to have it trashed instead of > the option of a recycler/reseller coming around and picking it up at no cost. > > As you said, it's just one of those things. The cables might still have some ultra-secret bits in them.
Re: AT&T routing problems towards www.worldspan.com?
> That host is not working for us either, but looks more like a host > problem rather then BGP problem. I have no problem getting to other > IP's in that range like 216.113.132.21 which is probably it's default > gateway. I can ping 216.113.132.21 from all the places I have tried too. So I agree with you, a possible host problem. *Or* it could be a problem involving parallell links with src/dst hashing where one of the links is not working properly. Steinar Haug, Nethelp consulting, sth...@nethelp.no
re: AT&T routing problems towards www.worldspan.com?
--- Original Message --- >From: sth...@nethelp.no >Sent: Monday, August 30, 2010 5:22 AM >To: nanog@nanog.org > >We have problems reaching www.worldspan.com (216.113.132.22) from >some locations. The common problem seems to be AT&T (AS 7018). Our >AS path towards the 216.113.128.0/19 prefix is typically > > 3356 7018 17228 19631 > >Anybody else see problems here? I note that I can ping 216.113.132.22 >from some locations within our network - but not, for instance, from >route-server.ip.att.net. > >Steinar Haug, Nethelp consulting, sth...@nethelp.no > Not sure if this would be helpful Similar from West Michigan-USA. I also have troubles via Qwest (USA) and Qwest (via Colt) in Germany. I too cannot ping that IP from these locations. My guess is a possible node issue, not network. #trace ip 216.113.132.22 Tracing the route to www.worldspan.net (216.113.132.22) 1 12.87.238.5 [AS 7018] 8 msec 4 msec 12 msec 2 cr82.dtrmi.ip.att.net (12.122.102.90) [AS 7018] 44 msec 40 msec 40 msec 11 12-122-254-82.attens.net (12.122.254.82) [AS 7018] 40 msec 40 msec 40 msec 12 mdf001c7613r0004-tge-12-1.atl1.attens.net (12.129.65.142) [AS 17228] 44 msec 44 msec 40 msec 13 12.129.64.122 [AS 17228] 40 msec 40 msec 12.129.64.126 [AS 17228] 40 msec 14 10.5.158.229 40 msec 10.5.158.225 44 msec 44 msec 15 * * * #sh ip bgp 216.113.132.22/19 BGP routing table entry for 216.113.128.0/19, version 54509208 Paths: (1 available, best #1, table Default-IP-Routing-Table) Advertised to update-groups: 1 7018 17228 19631 12.87.238.5 from 12.87.238.5 (12.122.83.244) Origin IGP, localpref 100, valid, external, best >From our German router (just a defgwy towards Qwest...) #trace ip 216.113.132.22 Tracing the route to 216.113.132.22 1 209.181.1.21 12 msec 12 msec 16 msec 2 65.120.25.53 16 msec 12 msec 16 msec 3 205.171.251.110 104 msec 104 msec 108 msec 4 192.205.34.253 108 msec 108 msec 108 msec 5 12.122.134.166 128 msec 124 msec 128 msec 6 12.122.1.173 120 msec 120 msec 120 msec 7 12.123.22.110 124 msec 128 msec 128 msec 8 12.122.141.69 124 msec 124 msec 128 msec 9 12.122.254.82 128 msec 120 msec 132 msec 10 12.129.65.142 124 msec 128 msec 12.129.65.138 120 msec 11 12.129.64.122 124 msec 12.129.64.126 120 msec 120 msec 12 * * * 13 * * * 14 * * *
Re: AT&T routing problems towards www.worldspan.com?
That host is not working for us either, but looks more like a host problem rather then BGP problem. I have no problem getting to other IP's in that range like 216.113.132.21 which is probably it's default gateway. On 08/30/2010 05:22 AM, sth...@nethelp.no wrote: We have problems reaching www.worldspan.com (216.113.132.22) from some locations. The common problem seems to be AT&T (AS 7018). Our AS path towards the 216.113.128.0/19 prefix is typically 3356 7018 17228 19631 Anybody else see problems here? I note that I can ping 216.113.132.22 from some locations within our network - but not, for instance, from route-server.ip.att.net. Steinar Haug, Nethelp consulting, sth...@nethelp.no
AT&T routing problems towards www.worldspan.com?
We have problems reaching www.worldspan.com (216.113.132.22) from some locations. The common problem seems to be AT&T (AS 7018). Our AS path towards the 216.113.128.0/19 prefix is typically 3356 7018 17228 19631 Anybody else see problems here? I note that I can ping 216.113.132.22 from some locations within our network - but not, for instance, from route-server.ip.att.net. Steinar Haug, Nethelp consulting, sth...@nethelp.no
Re: Did your BGP crash today?
Thomas, Wouldn't the confusion come from the fact that updates are considered as keepalives, so that Claudio sees so few type 4 messages because he receives updates ? Sec 4.2, Hold Time : "The calculated value indicates the maximum number of seconds that may elapse between the receipt of successive KEEPALIVE and/or UPDATE messages from the sender." Regards, Pierre. Thomas Mangin wrote: Apart from one big vendor most BGP speaker only send KEEPALIVES when they need to. So on my full feeds I see sessions running for more then 1 month which received less then 300 KEEPALIVE packets. The negociaged holdtime is always the lower value presented between two routers. The default HoldTime timer for Cisco is 180 seconds and for Juniper 90. So you should see a KEEPALIVE packet every minute from/to Cisco routers, and one every 30 seconds between Junipers. Should a BGP speaker do not see any KEEPALIVE during $HOLDTIME, it will tear the session down. You are telling me that your effective holdtime is 2592000 seconds when the HOLDTIME field is 16 bits ... hum ... http://www.faqs.org/rfcs/rfc4271.html section 4.2 So unless you know something I don't, I believe you are totally mistaken :) Thomas
Re: Did your BGP crash today?
> On Mon, 2010-08-30 at 10:58 +0200, Thomas Mangin wrote: >> http://www.faqs.org/rfcs/rfc4271.html section 4.2 >> >> So unless you know something I don't, I believe you are totally mistaken :) > > updates serve as implicit keepalives. Rule #1 do not post when you are not awake yet and quote the text which tells you are wrong .. broken :p Thank you Claudio for showing me why it would not work. Thomas
Re: Did your BGP crash today?
On Mon, 2010-08-30 at 10:58 +0200, Thomas Mangin wrote: > http://www.faqs.org/rfcs/rfc4271.html section 4.2 > > So unless you know something I don't, I believe you are totally mistaken :) updates serve as implicit keepalives. in that same section: "Hold Time: The calculated value indicates the maximum number of seconds that may elapse between the receipt of successive KEEPALIVE and/or UPDATE messages from the sender." also check section 6.5: "If a system does not receive successive KEEPALIVE, UPDATE, and/or NOTIFICATION messages [...]" --Daniel
Re: Did your BGP crash today?
> Apart from one big vendor most BGP speaker only send KEEPALIVES when they > need to. So on my full feeds I see sessions running for more then 1 month > which received less then 300 KEEPALIVE packets. The negociaged holdtime is always the lower value presented between two routers. The default HoldTime timer for Cisco is 180 seconds and for Juniper 90. So you should see a KEEPALIVE packet every minute from/to Cisco routers, and one every 30 seconds between Junipers. Should a BGP speaker do not see any KEEPALIVE during $HOLDTIME, it will tear the session down. You are telling me that your effective holdtime is 2592000 seconds when the HOLDTIME field is 16 bits ... hum ... http://www.faqs.org/rfcs/rfc4271.html section 4.2 So unless you know something I don't, I believe you are totally mistaken :) Thomas
Re: Did your BGP crash today?
On Sun, Aug 29, 2010 at 10:12:35PM +0200, Thomas Mangin wrote: > > It would seem to me that there should actually be a better option, e.g. > > recognizing the malformed update, and simply discarding it (and sending the > > originator an error message) instead of resetting the session. > > > > Resetting of BGP sessions should only be done in the most dire of > > circumstances, to avoid a widespread instability incident. > > > I had the same thought before giving up on it. > > Negotiating a new error message could be a per peer option. BGP has > capabilities for this exact reason. > > However to make sense you would need to find a resynchronisation point > to only exclude the one faulty message. Initially I thought that the > last received KEEPALIVE (for the receiver of the error message) could do > - but you find yourselves with races conditions - so perhaps two > KEEPALIVE back ? Apart from one big vendor most BGP speaker only send KEEPALIVES when they need to. So on my full feeds I see sessions running for more then 1 month which received less then 300 KEEPALIVE packets. -- :wq Claudio
