Re: Level 3, again
Just got notification that Level3 will be doing maintenance early AM EST to reboot an edge router in Miami, FL. The ticket # for this is 2365443 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Ranch Sent: Tuesday, February 12, 2008 1:32 PM To: 'David Hubbard'; nanog@merit.edu Subject: RE: Level 3, again This affected us too, same impact. No official word from Level3, but it looks to be back up now. My normal path was SJO-LAX-DAL-MIA, now it's SJO-LAX-DEN-DAL-MIA. I can only speculate that it was related to the LAX-DAL link. It lasted about an hour. Chris -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, February 12, 2008 10:06 AM To: nanog@merit.edu Subject: Level 3, again Anyone know what's going on on their network? We opened a ticket but haven't heard back, sounds like they may have some kind of nationwide issue going on that started in Atlanta. We've had customers on Time Warner, SBC Global, ATT and Pac Bell unable to reach us. David
RE: Level 3, again
This affected us too, same impact. No official word from Level3, but it looks to be back up now. My normal path was SJO-LAX-DAL-MIA, now it's SJO-LAX-DEN-DAL-MIA. I can only speculate that it was related to the LAX-DAL link. It lasted about an hour. Chris -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, February 12, 2008 10:06 AM To: nanog@merit.edu Subject: Level 3, again Anyone know what's going on on their network? We opened a ticket but haven't heard back, sounds like they may have some kind of nationwide issue going on that started in Atlanta. We've had customers on Time Warner, SBC Global, ATT and Pac Bell unable to reach us. David
Level 3, again
Anyone know what's going on on their network? We opened a ticket but haven't heard back, sounds like they may have some kind of nationwide issue going on that started in Atlanta. We've had customers on Time Warner, SBC Global, ATT and Pac Bell unable to reach us. David
RE: Level 3, again
For what it's worth, from Canada I can get to Disney.com and theplanet.com via Level(3) no problem. Paul -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, February 12, 2008 1:06 PM To: nanog@merit.edu Subject: Level 3, again Anyone know what's going on on their network? We opened a ticket but haven't heard back, sounds like they may have some kind of nationwide issue going on that started in Atlanta. We've had customers on Time Warner, SBC Global, ATT and Pac Bell unable to reach us. David
Re: Level 3, again
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 fyi -/ - Original Message Subject: [Outages] FW: Level(3) Service Affecting Maintenance - GCR#180226 -Scheduled Date: Tue, 12 Feb 2008 13:22:18 -0700 From: Raman Sud [EMAIL PROTECTED] To: [EMAIL PROTECTED] [EMAIL PROTECTED] Level (3) Communications - Network Change Control (NCC) ***SCHEDULED MAINTENANCE ADVISEMENT*** Level(3) Service Affecting Maintenance - GCR#180226 - Scheduled Clarify Case#: 2365443 GCR #: CHG00180226 Primary Dates: 13 Feb 2008 01:00:00 - 13 Feb 2008 02:00:00 Eastern Primary Dates GMT: 13 Feb 2008 06:00:00 - 13 Feb 2008 07:00:00 Location of Maintenance: Miami FL, Summary Description of Maintenance: Reboot - Router - Edge Router Customer Impact: Service Expected Impact Classification Duration Additional Notes IP - Redundant Outage SA Up to 30 minutes DEMAND MAINTENANCE: Level 3 will be rebooting router due to bogus cells incrementing. IP - Non-Redundant Outage SA Up to 30 minutes DEMAND MAINTENANCE: Level 3 will be rebooting router due to bogus cells incrementing. ___ Outages mailing list [EMAIL PROTECTED] http://isotf.org/mailman/listinfo/outages regards, /virendra Edward A. Trdina III wrote: Just got notification that Level3 will be doing maintenance early AM EST to reboot an edge router in Miami, FL. The ticket # for this is 2365443 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Ranch Sent: Tuesday, February 12, 2008 1:32 PM To: 'David Hubbard'; nanog@merit.edu Subject: RE: Level 3, again This affected us too, same impact. No official word from Level3, but it looks to be back up now. My normal path was SJO-LAX-DAL-MIA, now it's SJO-LAX-DEN-DAL-MIA. I can only speculate that it was related to the LAX-DAL link. It lasted about an hour. Chris -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, February 12, 2008 10:06 AM To: nanog@merit.edu Subject: Level 3, again Anyone know what's going on on their network? We opened a ticket but haven't heard back, sounds like they may have some kind of nationwide issue going on that started in Atlanta. We've had customers on Time Warner, SBC Global, ATT and Pac Bell unable to reach us. David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHshf5pbZvCIJx1bcRAhF4AJ0cIGH8yMgD6o51pdnFMqA8FWja+wCeLDvq Eyf/Zce2zgnO08GpYGQ78+8= =ak1G -END PGP SIGNATURE-
RE: Level 3, again
Just got confirmation from them that a master ticket is being created and the transport team is looking at it. There's an issue with an EBR router in Dallas that they have already routed around but from what I can see, there are still major issues with traffic that has to traverse Dallas. Our connection to L3 in Atlanta going to the same destinations has no issue but jumps from ATL to DC to LAX. David -Original Message- From: Chris Ranch [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 12, 2008 1:32 PM To: David Hubbard; nanog@merit.edu Subject: RE: Level 3, again This affected us too, same impact. No official word from Level3, but it looks to be back up now. My normal path was SJO-LAX-DAL-MIA, now it's SJO-LAX-DEN-DAL-MIA. I can only speculate that it was related to the LAX-DAL link. It lasted about an hour. Chris -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, February 12, 2008 10:06 AM To: nanog@merit.edu Subject: Level 3, again Anyone know what's going on on their network? We opened a ticket but haven't heard back, sounds like they may have some kind of nationwide issue going on that started in Atlanta. We've had customers on Time Warner, SBC Global, ATT and Pac Bell unable to reach us. David
Level 3 (3356) issues?
Just curious if anyone is seeing issues with Level 3 right now? Our session is still up but we can't see any outside routes through them currently. I'm guessing by the fact that I've been on hold for 25 minutes that I'm not the only one having an issue with them but wanted to double check. Thanks, David
RE: Level 3 (3356) issues?
No issues here full feed coming in and no issues getting out (that have been noticed so far) 2 so-8-0.hsa1.Detroit1.Level3.net (166.90.248.1) [AS 3356] 12 msec 8 msec 12 msec 3 so-4-3-0.mp1.Detroit1.Level3.net (4.68.115.1) [AS 3356] 12 msec 12 msec 8 msec 4 as-4-0.bbr2.NewYork1.Level3.net (64.159.0.238) [AS 3356] 36 msec ae-0-0.bbr1.NewYork1.Level3.net (64.159.1.41) [AS 3356] 40 msec 36 msec 5 ae-4-99.edge6.NewYork1.Level3.net (4.68.16.202) [AS 3356] 36 msec ae-3-89.edge6.NewYork1.Level3.net (4.68.16.138) [AS 3356] 36 msec ae-1-69.edge6.NewYork1.Level3.net (4.68.16.10) [AS 3356] 36 msec 6 pop2-nye-P5-0.atdn.net (66.185.137.209) [AS 1668] 40 msec 40 msec 36 msec 7 bb1-nye-P1-0.atdn.net (66.185.151.64) [AS 1668] 40 msec 36 msec 40 msec 8 bb2-ash-P13-0.atdn.net (66.185.152.87) [AS 1668] 36 msec 36 msec 40 msec 9 pop1-ash-S1-1-0.atdn.net (66.185.144.35) [AS 1668] 40 msec 36 msec 36 msec 10 dar1-mtc-S0-0-0.atdn.net (66.185.148.222) [AS 1668] 40 msec dar1-mtc-S1-2-0.atdn.net (66.185.152.105) [AS 1668] 40 msec 40 msec Take care, Paul Stewart Senior Network Administrator Nexicom 5 King St. E., Millbrook, ON, LOA 1GO Phone: 705-932-4127 Web: http://www.nexicom.net Nexicom - Connected. Naturally. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, January 15, 2008 11:45 AM To: nanog@merit.edu Subject: Level 3 (3356) issues? Just curious if anyone is seeing issues with Level 3 right now? Our session is still up but we can't see any outside routes through them currently. I'm guessing by the fact that I've been on hold for 25 minutes that I'm not the only one having an issue with them but wanted to double check. Thanks, David The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you.
Re: Level 3 (3356) issues?
Our DS3 here in Cupertino, Ca seems to be working flawless -Mike On Jan 15, 2008 8:44 AM, David Hubbard [EMAIL PROTECTED] wrote: Just curious if anyone is seeing issues with Level 3 right now? Our session is still up but we can't see any outside routes through them currently. I'm guessing by the fact that I've been on hold for 25 minutes that I'm not the only one having an issue with them but wanted to double check. Thanks, David
RE: Level 3 (3356) issues?
Looks like this was localized to Tampa. I've received emails from two other people connected through Tampa, like us, who were having the same issues. I finally got TCAM on the phone after about an hour. They have a master ticket for a failure of three DLM modules lasting 47 minutes but it is showing believed to be resolved via reset but they have not yet diagnosed the cause. David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Hubbard Sent: Tuesday, January 15, 2008 11:45 AM To: nanog@merit.edu Subject: Level 3 (3356) issues? Just curious if anyone is seeing issues with Level 3 right now? Our session is still up but we can't see any outside routes through them currently. I'm guessing by the fact that I've been on hold for 25 minutes that I'm not the only one having an issue with them but wanted to double check. Thanks, David
Re: Level 3 (3356) issues?
I know of Level 3 issues in the Tampa area. Where are you? -Scott - Original Message - From: David Hubbard [EMAIL PROTECTED] To: nanog@merit.edu Sent: Tuesday, January 15, 2008 11:44:31 AM (GMT-0500) America/New_York Subject: Level 3 (3356) issues? Just curious if anyone is seeing issues with Level 3 right now? Our session is still up but we can't see any outside routes through them currently. I'm guessing by the fact that I've been on hold for 25 minutes that I'm not the only one having an issue with them but wanted to double check. Thanks, David
RE: Level 3 in Ohio
Level3 was experiencing problems with their own 8xx number yesterday. Start Date/Time: 9/19/2007 17:30 GMT End Date/Time: Ongoing Summary: Level 3 Communications is currently investigating issues with calls to our Toll Free 8774538353. If you are attempting to call 8774538353 and get fast busy please send e-mail to [EMAIL PROTECTED] A detailed update will shortly follow informing you of the results of the investigation. Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Marshall Eubanks Sent: Wednesday, September 19, 2007 1:34 PM To: nanog list Subject: Level 3 in Ohio Is anyone reporting Level3 outages in Ohio or DC ? One of my clients is down, and L3 is not answering the phones (!) traceroute 65.89.42.1 (From Cogent in Tyson's Corner) traceroute to 65.89.42.1 (65.89.42.1), 30 hops max, 38 byte packets 1 dmz-mct2.multicasttech.com (63.105.122.1) 0.367 ms 0.265 ms 0.238 ms 2 g0-7.na21.b002176-1.dca01.atlas.cogentco.com (38.99.206.153) 0.598 ms 0.548 ms 0.529 ms 3 g2-1-3587.core01.dca01.atlas.cogentco.com (38.20.32.13) 0.862 ms 0.834 ms 0.740 ms 4 t4-1.mpd01.dca01.atlas.cogentco.com (154.54.3.158) 0.801 ms 0.879 ms * 5 v3493.mpd01.dca02.atlas.cogentco.com (154.54.7.2) 1.328 ms 1.311 ms 1.247 ms 6 t1-4.mpd01.iad01.atlas.cogentco.com (154.54.7.162) 1.693 ms 1.598 ms 1.605 ms 7 g4-0-3490.core01.iad01.atlas.cogentco.com (154.54.3.225) 1.411 ms 1.453 ms 1.552 ms 8 oc48-6-0-2.edge2.Washington3.Level3.net (4.68.127.9) 1.577 ms 1.588 ms 16.498 ms 9 ae-44-99.car4.Washington1.Level3.net (4.68.17.198) 2.766 ms ae-24-79.car4.Washington1.Level3.net (4.68.17.70) 3.282 ms ae-34-89.car4.Washington1.Level3.net (4.68.17.134) 2.808 ms time out Cox Cable in Northern Virginia [TME-Laptop-2:~/Movies/NoisiVision] tme% traceroute 65.89.42.1 traceroute to 65.89.42.1 (65.89.42.1), 64 hops max, 40 byte packets 1 * * * 2 ip70-179-104-1.dc.dc.cox.net (70.179.104.1) 14.989 ms 11.563 ms 11.782 ms 3 ip68-100-1-161.dc.dc.cox.net (68.100.1.161) 18.078 ms 12.329 ms 12.036 ms 4 ip68-100-0-1.dc.dc.cox.net (68.100.0.1) 13.368 ms 12.301 ms 11.960 ms 5 mrfddsrj01-ge110.rd.dc.cox.net (68.100.0.161) 12.504 ms 11.729 ms * 6 xe-9-2-0.edge1.washington1.level3.net (4.79.228.61) 59.189 ms 12.721 ms 11.749 ms 7 ae-44-99.car4.washington1.level3.net (4.68.17.198) 13.502 ms 13.389 ms ae-34-89.car4.washington1.level3.net (4.68.17.134) 14.290 ms time out (Note that both trace routes appear to be flapping at the last reported hop. Regards Marshall
Level 3 in Ohio
Is anyone reporting Level3 outages in Ohio or DC ? One of my clients is down, and L3 is not answering the phones (!) traceroute 65.89.42.1 (From Cogent in Tyson's Corner) traceroute to 65.89.42.1 (65.89.42.1), 30 hops max, 38 byte packets 1 dmz-mct2.multicasttech.com (63.105.122.1) 0.367 ms 0.265 ms 0.238 ms 2 g0-7.na21.b002176-1.dca01.atlas.cogentco.com (38.99.206.153) 0.598 ms 0.548 ms 0.529 ms 3 g2-1-3587.core01.dca01.atlas.cogentco.com (38.20.32.13) 0.862 ms 0.834 ms 0.740 ms 4 t4-1.mpd01.dca01.atlas.cogentco.com (154.54.3.158) 0.801 ms 0.879 ms * 5 v3493.mpd01.dca02.atlas.cogentco.com (154.54.7.2) 1.328 ms 1.311 ms 1.247 ms 6 t1-4.mpd01.iad01.atlas.cogentco.com (154.54.7.162) 1.693 ms 1.598 ms 1.605 ms 7 g4-0-3490.core01.iad01.atlas.cogentco.com (154.54.3.225) 1.411 ms 1.453 ms 1.552 ms 8 oc48-6-0-2.edge2.Washington3.Level3.net (4.68.127.9) 1.577 ms 1.588 ms 16.498 ms 9 ae-44-99.car4.Washington1.Level3.net (4.68.17.198) 2.766 ms ae-24-79.car4.Washington1.Level3.net (4.68.17.70) 3.282 ms ae-34-89.car4.Washington1.Level3.net (4.68.17.134) 2.808 ms time out Cox Cable in Northern Virginia [TME-Laptop-2:~/Movies/NoisiVision] tme% traceroute 65.89.42.1 traceroute to 65.89.42.1 (65.89.42.1), 64 hops max, 40 byte packets 1 * * * 2 ip70-179-104-1.dc.dc.cox.net (70.179.104.1) 14.989 ms 11.563 ms 11.782 ms 3 ip68-100-1-161.dc.dc.cox.net (68.100.1.161) 18.078 ms 12.329 ms 12.036 ms 4 ip68-100-0-1.dc.dc.cox.net (68.100.0.1) 13.368 ms 12.301 ms 11.960 ms 5 mrfddsrj01-ge110.rd.dc.cox.net (68.100.0.161) 12.504 ms 11.729 ms * 6 xe-9-2-0.edge1.washington1.level3.net (4.79.228.61) 59.189 ms 12.721 ms 11.749 ms 7 ae-44-99.car4.washington1.level3.net (4.68.17.198) 13.502 ms 13.389 ms ae-34-89.car4.washington1.level3.net (4.68.17.134) 14.290 ms time out (Note that both trace routes appear to be flapping at the last reported hop. Regards Marshall
Re: Level 3 in Ohio
It's back up in Ohio, as of 2:05 PM EDT. On Sep 19, 2007, at 1:33 PM, Marshall Eubanks wrote: Is anyone reporting Level3 outages in Ohio or DC ? One of my clients is down, and L3 is not answering the phones (!) traceroute 65.89.42.1 (From Cogent in Tyson's Corner) traceroute to 65.89.42.1 (65.89.42.1), 30 hops max, 38 byte packets 1 dmz-mct2.multicasttech.com (63.105.122.1) 0.367 ms 0.265 ms 0.238 ms 2 g0-7.na21.b002176-1.dca01.atlas.cogentco.com (38.99.206.153) 0.598 ms 0.548 ms 0.529 ms 3 g2-1-3587.core01.dca01.atlas.cogentco.com (38.20.32.13) 0.862 ms 0.834 ms 0.740 ms 4 t4-1.mpd01.dca01.atlas.cogentco.com (154.54.3.158) 0.801 ms 0.879 ms * 5 v3493.mpd01.dca02.atlas.cogentco.com (154.54.7.2) 1.328 ms 1.311 ms 1.247 ms 6 t1-4.mpd01.iad01.atlas.cogentco.com (154.54.7.162) 1.693 ms 1.598 ms 1.605 ms 7 g4-0-3490.core01.iad01.atlas.cogentco.com (154.54.3.225) 1.411 ms 1.453 ms 1.552 ms 8 oc48-6-0-2.edge2.Washington3.Level3.net (4.68.127.9) 1.577 ms 1.588 ms 16.498 ms 9 ae-44-99.car4.Washington1.Level3.net (4.68.17.198) 2.766 ms ae-24-79.car4.Washington1.Level3.net (4.68.17.70) 3.282 ms ae-34-89.car4.Washington1.Level3.net (4.68.17.134) 2.808 ms time out Cox Cable in Northern Virginia [TME-Laptop-2:~/Movies/NoisiVision] tme% traceroute 65.89.42.1 traceroute to 65.89.42.1 (65.89.42.1), 64 hops max, 40 byte packets 1 * * * 2 ip70-179-104-1.dc.dc.cox.net (70.179.104.1) 14.989 ms 11.563 ms 11.782 ms 3 ip68-100-1-161.dc.dc.cox.net (68.100.1.161) 18.078 ms 12.329 ms 12.036 ms 4 ip68-100-0-1.dc.dc.cox.net (68.100.0.1) 13.368 ms 12.301 ms 11.960 ms 5 mrfddsrj01-ge110.rd.dc.cox.net (68.100.0.161) 12.504 ms 11.729 ms * 6 xe-9-2-0.edge1.washington1.level3.net (4.79.228.61) 59.189 ms 12.721 ms 11.749 ms 7 ae-44-99.car4.washington1.level3.net (4.68.17.198) 13.502 ms 13.389 ms ae-34-89.car4.washington1.level3.net (4.68.17.134) 14.290 ms time out (Note that both trace routes appear to be flapping at the last reported hop. Regards Marshall
RE: Level 3 in Ohio
I can ping 65.89.42.1 from here and it seems to be going through level3. traceroute to 65.89.42.1 (65.89.42.1), 30 hops max, 38 byte packets 1 wookie-02.core..net () 0.454 ms 0.458 ms 0.320 ms 2 wookie-01-fe-2-0.core..net (xx) 0.678 ms 0.648 ms 0.559 ms 3 ge-5-0-102.hsa2.Cincinnati1.Level3.net (63.210.xx) 0.826 ms 0.747 ms 1.742 ms 4 so-5-0-0.mpls1.Cincinnati1.Level3.net (4.68.124.241) 0.819 ms 0.858 ms 1.102 ms 5 so-2-0-1.bbr2.Chicago1.Level3.net (64.159.0.162) 7.157 ms 7.216 ms ae-0-0.bbr1.Chicago1.Level3.net (64.159.1.33) 6.998 ms 6 ae-24-52.car4.Chicago1.Level3.net (4.68.101.40) 7.449 ms ae-14-51.car4.Chicago1.Level3.net (4.68.101.8) 7.525 ms ae-14-53.car4.Chicago1.Level3.net (4.68.101.72) 7.503 ms 7 te-4-1-73.rp0-5.chcgilca.Level3.net (4.68.63.14) 8.883 ms 11.020 ms 8.155 ms 8 P5-0.a0.chcg.broadwing.net (216.140.14.109) 8.120 ms 13.737 ms 8.424 ms 9 p2-0-0.e1.chcg.broadwing.net (216.140.15.30) 8.987 ms 7.944 ms 8.161 ms 10 67.99.9.158 (67.99.9.158) 15.497 ms * 16.167 ms Mike Walter, MCP Systems Administrator 3z.net a PCD Company http://www.3z.net When Success is the Only Solution think 3z.net tel: 859.331.9004 fax: 859.578.3522 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Marshall Eubanks Sent: Wednesday, September 19, 2007 1:34 PM To: nanog list Subject: Level 3 in Ohio Is anyone reporting Level3 outages in Ohio or DC ? One of my clients is down, and L3 is not answering the phones (!) traceroute 65.89.42.1 (From Cogent in Tyson's Corner) traceroute to 65.89.42.1 (65.89.42.1), 30 hops max, 38 byte packets 1 dmz-mct2.multicasttech.com (63.105.122.1) 0.367 ms 0.265 ms 0.238 ms 2 g0-7.na21.b002176-1.dca01.atlas.cogentco.com (38.99.206.153) 0.598 ms 0.548 ms 0.529 ms 3 g2-1-3587.core01.dca01.atlas.cogentco.com (38.20.32.13) 0.862 ms 0.834 ms 0.740 ms 4 t4-1.mpd01.dca01.atlas.cogentco.com (154.54.3.158) 0.801 ms 0.879 ms * 5 v3493.mpd01.dca02.atlas.cogentco.com (154.54.7.2) 1.328 ms 1.311 ms 1.247 ms 6 t1-4.mpd01.iad01.atlas.cogentco.com (154.54.7.162) 1.693 ms 1.598 ms 1.605 ms 7 g4-0-3490.core01.iad01.atlas.cogentco.com (154.54.3.225) 1.411 ms 1.453 ms 1.552 ms 8 oc48-6-0-2.edge2.Washington3.Level3.net (4.68.127.9) 1.577 ms 1.588 ms 16.498 ms 9 ae-44-99.car4.Washington1.Level3.net (4.68.17.198) 2.766 ms ae-24-79.car4.Washington1.Level3.net (4.68.17.70) 3.282 ms ae-34-89.car4.Washington1.Level3.net (4.68.17.134) 2.808 ms time out Cox Cable in Northern Virginia [TME-Laptop-2:~/Movies/NoisiVision] tme% traceroute 65.89.42.1 traceroute to 65.89.42.1 (65.89.42.1), 64 hops max, 40 byte packets 1 * * * 2 ip70-179-104-1.dc.dc.cox.net (70.179.104.1) 14.989 ms 11.563 ms 11.782 ms 3 ip68-100-1-161.dc.dc.cox.net (68.100.1.161) 18.078 ms 12.329 ms 12.036 ms 4 ip68-100-0-1.dc.dc.cox.net (68.100.0.1) 13.368 ms 12.301 ms 11.960 ms 5 mrfddsrj01-ge110.rd.dc.cox.net (68.100.0.161) 12.504 ms 11.729 ms * 6 xe-9-2-0.edge1.washington1.level3.net (4.79.228.61) 59.189 ms 12.721 ms 11.749 ms 7 ae-44-99.car4.washington1.level3.net (4.68.17.198) 13.502 ms 13.389 ms ae-34-89.car4.washington1.level3.net (4.68.17.134) 14.290 ms time out (Note that both trace routes appear to be flapping at the last reported hop. Regards Marshall
Level 3 Colo question
Question for you all; does anyone have experience with Level 3's colo offerings, and if so, have your prices increased dramatically for power and square footage as contract renewals have come around? Do you know if they have a practice of pricing out customers they'd prefer to have move elsewhere? Would like to compare notes privately if so... Thanks, David
Re: Level(3) faux paux
On Wed, 11 Jul 2007, Security Admin (NetSec) wrote: I have noticed that Level(3) misconfigs/outages seem to happen more frequently than with most other Tier 1's. Am unsure whether or not this maybe they have a larger change-rate? or more folks that notice problems and complain here? (note that I don't know but suspect everyone has a relatively close approximation of this figure at a certain place in the network-size-tiering) show up more often with issues than say Sprint [AS1239]. Is their any (cause they don't let Ted on routers anymore...) one (or any coporation) that keeps track of outages such as these? Would think it might be a good thing to know for proper mulit-homing relationships to minimize the type of outage that Yahoo faced... Because someone 3 as-hops away sucking down your prefix and traffic is your direct provider's problem how? -Chris
RE: Level(3) faux paux
Maybe level 3 have more vocal or higher profile customer base? Personally speaking I've found level 3 to have outstanding service and network availibility. Neil. -Original Message- From: Security Admin (NetSec) [EMAIL PROTECTED] Sent: 12 July 2007 06:56 To: nanog@merit.edu nanog@merit.edu Subject: Level(3) faux paux I have noticed that Level(3) misconfigs/outages seem to happen more frequently than with most other Tier 1s. Am unsure whether or not this is a mis-statement, but based on NANOG posts, Level(3) [AS3356] seems to show up more often with issues than say Sprint [AS1239]. Is their any one (or any coporation) that keeps track of outages such as these? Would think it might be a good thing to know for proper mulit-homing relationships to minimize the type of outage that Yahoo faced SecAdmin (aka Edward Ray)
Re: Level(3) faux paux
Cuz they are taking over at a rate that their best engineers can¹t meet. On 7/11/07 11:08 PM, Chris L. Morrow [EMAIL PROTECTED] wrote: On Wed, 11 Jul 2007, Security Admin (NetSec) wrote: I have noticed that Level(3) misconfigs/outages seem to happen more frequently than with most other Tier 1's. Am unsure whether or not this maybe they have a larger change-rate? or more folks that notice problems and complain here? (note that I don't know but suspect everyone has a relatively close approximation of this figure at a certain place in the network-size-tiering) show up more often with issues than say Sprint [AS1239]. Is their any (cause they don't let Ted on routers anymore...) one (or any coporation) that keeps track of outages such as these? Would think it might be a good thing to know for proper mulit-homing relationships to minimize the type of outage that Yahoo faced... Because someone 3 as-hops away sucking down your prefix and traffic is your direct provider's problem how? -Chris
Re: Level(3) faux paux
[EMAIL PROTECTED] wrote on 07/12/2007 01:56:32 AM: I have noticed that Level(3) misconfigs/outages seem to happen more frequently than with most other Tier 1?s. Am unsure whether or not this is a mis-statement, but based on NANOG posts, Level(3) [AS3356] seems to show up more often with issues than say Sprint [AS1239]. As mentioned previously, there are a lot of variables that effect this claim. Is their any one (or any coporation) that keeps track of outages such as these? Would think it might be a good thing to know for proper mulit-homing relationships to minimize the type of outage that Yahoo faced? A lot of businesses include this sort of data in their quarterly reports. Some of your questions might be answered by reports like this one = http://www.level3.com/newsroom/pressreleases/2007/20070426.html Additionally, I'd do a Google search for terms like System Availability and include some Tier 1 providers to see what kind of data comes back. Then just go from there. Best regards, Tim Rainier _ THIS E-MAIL is private correspondence and is intended only for the identified recipients. We attempt to correctly address all e-mails, but if for any reason you have received this message in error, please take notice that you should not disclose or distribute this message to any other person. You should immediately notify the sender and delete this message. If the message contains or attaches CONFIDENTIAL information, you must treat that information confidentially. For questions, please contact the sender.
Re: Level(3) faux paux
On Wed, 11 Jul 2007 22:56:32 PDT, Security Admin (NetSec) said: Am unsure whether or not this is a mis-statement, but based on NANOG posts, Level(3) [AS3356] seems to show up mor=e often with issues than say Sprint [AS1239]. How many places does AS3356 connect with other AS's, and how many places does AS1239 connec with other stuff? I'd expects an AS with 500 interchange points would have 25% more whoopsies than one with 400 interchanges even if they were otherwise equivalent. Another factor would be number of miles of fiber their net is run over - If backhoes per mile is a constant, a 3,000 mile link is more likely to be hit than one half as long, and so on. Or maybe Level3's network is more openly visible from outside, so it's easier to tell that they are the source of a problem than a net that's not easily debugged from outside the AS (leaving you wondering if it's them or somebody on the other side of them). Or maybe past experience has shown that the two have the same *actual* failure rate, but asking for a Level3 help is more likely to actually get you a clueful *and* helpful engineer. Plus, I don't think *any* provider gets mentioned enough on NANOG to be able to draw any realistic statistical inferences. The short-form highly inaccurate handwave is that you'd need *two* providers and at least 20-30 datapoints on *each* to draw conclusions - which would probably take you 2-3 years to collect, and at that point, changing management policies at both providers will render your results inaccurate (you'd be comparing 3-year old data to current). pgpobYv3oYZqN.pgp Description: PGP signature
RE: Level(3) faux paux
From: [EMAIL PROTECTED] Or maybe past experience has shown that the two have the same *actual* failure rate, but asking for a Level3 help is more likely to actually get you a clueful *and* helpful engineer. This has been my experience when we've had issues with any of the five companies we purchase from; Level 3 has consistently been able to put someone on the phone that actually knows how to resolve problems without even needing to call me back; most of the others we turn our link off because I know it's going to be several hours. David
Level(3) faux paux
I have noticed that Level(3) misconfigs/outages seem to happen more frequently than with most other Tier 1's. Am unsure whether or not this is a mis-statement, but based on NANOG posts, Level(3) [AS3356] seems to show up more often with issues than say Sprint [AS1239]. Is their any one (or any coporation) that keeps track of outages such as these? Would think it might be a good thing to know for proper mulit-homing relationships to minimize the type of outage that Yahoo faced... SecAdmin (aka Edward Ray) -- This mail was scanned by BitDefender For more informations please visit http://www.bitdefender.com
power problems yesterday (3/26/07) at Level 3 in Atlanta
I'm hearing from one of our carriers about a power problem at Level 3's Courtland Street facility in Atlanta yesterday that took them completely off-line. Anyone else hear or know anything about this event, if and when it happened? Any details? Regards, -Dorn
Re: power problems yesterday (3/26/07) at Level 3 in Atlanta
The problem was reported to be at the Courtland Street gateway. The folks that reported it to me are on AC power and stated that both their A and B feeds went dark. I have an Asterisk/IAX connection with one of their boxes there that reported about 30 minutes of no contact yesterday afternoon. I don't know about DC power but suspect it much less likely to have gone down. Still, it must have been something unusual to take out both A and B feeds of UPS AC... On 3/27/07, Todd White [EMAIL PROTECTED] wrote: We are in the old Wiltel facility in Norcross now a L3 facility and had no issues. If you get some good answers directly off the list let me know as we are looking to possibly move downtown. thanks -Todd On 3/27/07, Dorn Hetzel [EMAIL PROTECTED] wrote: I'm hearing from one of our carriers about a power problem at Level 3's Courtland Street facility in Atlanta yesterday that took them completely off-line. Anyone else hear or know anything about this event, if and when it happened? Any details? Regards, -Dorn
RE: power problems yesterday (3/26/07) at Level 3 in Atlanta
I can confirm there was a power issue at ATL1 yesterday at about 2:15PM EST. What I was told was that there was a main breaker that flipped that may have been caused by a surge, but they never went to generators. I have no official details beyond that but they were down for about 30 minutes from what I could see. -Scott From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dorn Hetzel Sent: Tuesday, March 27, 2007 9:30 AM To: nanog@merit.edu Subject: power problems yesterday (3/26/07) at Level 3 in Atlanta I'm hearing from one of our carriers about a power problem at Level 3's Courtland Street facility in Atlanta yesterday that took them completely off-line. Anyone else hear or know anything about this event, if and when it happened? Any details? Regards, -Dorn
Re: SaidCom disconnected by Level 3 (former Telcove property)
On Sat, 17 Mar 2007, Brandon Galbraith wrote: True :) I'd also think (read: hope) if an organization was located in an area where multi-homing was not possible, then that organization and its customers would not be doing things that are mission critical, i.e. business stops if there is no Internet connectivity. Mission critical seems to be quite subjective these days. Sure. That's why I qualified my remarks. jms
RE: SaidCom disconnected by Level 3 (former Telcove property)
Almost ALL providers should be multihomed. --Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of virendra rode // Sent: Thursday, March 15, 2007 11:26 AM To: NANOG Subject: Re: SaidCom disconnected by Level 3 (former Telcove property) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank - Just curious, should small responsive providers should be multi-homed? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+XOApbZvCIJx1bcRAtkwAJ9vNak3F8FlCf9VDycf6IlAr445nACg59kB w2OWAGdchd2XQyxxgZWQaug= =Yb1+ -END PGP SIGNATURE-
Re: SaidCom disconnected by Level 3 (former Telcove property)
Almost ALL? Any company, or any person for that matter, that relies on their Internet connectivity for their lively hood should be multihomed. -wil On Mar 16, 2007, at 4:42 PM, Mike Hammett wrote: Almost ALL providers should be multihomed. --Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of virendra rode // Sent: Thursday, March 15, 2007 11:26 AM To: NANOG Subject: Re: SaidCom disconnected by Level 3 (former Telcove property) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank - Just curious, should small responsive providers should be multi- homed? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+XOApbZvCIJx1bcRAtkwAJ9vNak3F8FlCf9VDycf6IlAr445nACg59kB w2OWAGdchd2XQyxxgZWQaug= =Yb1+ -END PGP SIGNATURE-
RE: SaidCom disconnected by Level 3 (former Telcove property)
Some locations are just too cost prohibitive to multihome, but that really is a select few. Few places are out of the reach of a couple wireless hops back to civilization. --Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Wil Schultz Sent: Friday, March 16, 2007 6:56 PM To: nanog@merit.edu Subject: Re: SaidCom disconnected by Level 3 (former Telcove property) Almost ALL? Any company, or any person for that matter, that relies on their Internet connectivity for their lively hood should be multihomed. -wil On Mar 16, 2007, at 4:42 PM, Mike Hammett wrote: Almost ALL providers should be multihomed. --Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of virendra rode // Sent: Thursday, March 15, 2007 11:26 AM To: NANOG Subject: Re: SaidCom disconnected by Level 3 (former Telcove property) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank - Just curious, should small responsive providers should be multi- homed? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+XOApbZvCIJx1bcRAtkwAJ9vNak3F8FlCf9VDycf6IlAr445nACg59kB w2OWAGdchd2XQyxxgZWQaug= =Yb1+ -END PGP SIGNATURE-
Re: SaidCom disconnected by Level 3 (former Telcove property)
On 16-Mar-2007, at 19:56, Wil Schultz wrote: Almost ALL? Surely all those except those who are competing with you for the same customers should multi-home. :-) Joe
Re: SaidCom disconnected by Level 3 (former Telcove property)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 16, 2007, at 6:59 PM, Mike Hammett wrote: Some locations are just too cost prohibitive to multihome, but that really is a select few. It isn't just cost but can be path diversity (or lack thereof). We used to be headquartered 210 miles from civilization. We had a choice of providers and could have multihomed. However, the only realistic way for any of those providers to get to us would have been Bell frame relay. Since by far the most likely point of failure was the last mile (which was 210 miles), we made a decision that actually multihoming wasn't a good use of resources. We instead went with a good quality regional provider who was themselves multihomed. Now clearly there were cases where that wouldn't have any good but given the remoteness it just seems most likely that anything that took out one provider would have taken the other one as well. Now this case we are discussing is probably the exception to our assumptions but we had a much better provider at the time than Level3 ;-] From the sounds of the original post I wouldn't be too surprised if it was also fairly remote. Chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iD8DBQFF+zOWElUlCLUT2d0RAtwLAJ9esOECOSbeXOpPhPbEL3A9vmbJ5wCfWgnU Dd4lEmIoaMtPCRU9WXJRSVo= =wxdX -END PGP SIGNATURE-
RE: SaidCom disconnected by Level 3 (former Telcove property)
I've been working at a smaller ISP (~4000 subs, plus businesses), and not one has asked me if I'm multi-homed. When we or our upstream provider have a problem the telephones light up and people act as if it's a problem, but the reality is that they're not communicating it, up front, as a business requirement. Frank -Original Message- Sent: Friday, March 16, 2007 8:54 PM To: nanog@merit.edu Subject: Re: SaidCom disconnected by Level 3 (former Telcove property) Almost ALL? Any company, or any person for that matter, that relies on their Internet connectivity for their lively hood should be multihomed. -wil On Mar 16, 2007, at 4:42 PM, Mike Hammett wrote: Almost ALL providers should be multihomed. --Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of virendra rode // Sent: Thursday, March 15, 2007 11:26 AM To: NANOG Subject: Re: SaidCom disconnected by Level 3 (former Telcove property) -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank - Just curious, should small responsive providers should be multi- homed? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+XOApbZvCIJx1bcRAtkwAJ9vNak3F8FlCf9VDycf6IlAr445nACg59kB w2OWAGdchd2XQyxxgZWQaug= =Yb1+ -END PGP SIGNATURE-
Re: SaidCom disconnected by Level 3 (former Telcove property)
Joe Abley wrote: On 16-Mar-2007, at 19:56, Wil Schultz wrote: Almost ALL? Surely all those except those who are competing with you for the same customers should multi-home. :-) To the NANOG T-shirt Committee: Please consider this as the slogan for the next design.
Re: SaidCom disconnected by Level 3 (former Telcove property)
On 3/16/07, Justin M. Streiner [EMAIL PROTECTED] wrote: On Fri, 16 Mar 2007, Joe Abley wrote: Almost ALL? Surely all those except those who are competing with you for the same customers should multi-home. :-) True :) I'd also think (read: hope) if an organization was located in an area where multi-homing was not possible, then that organization and its customers would not be doing things that are mission critical, i.e. business stops if there is no Internet connectivity. Mission critical seems to be quite subjective these days. -brandon
Re: SaidCom disconnected by Level 3 (former Telcove property)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank - Just curious, should small responsive providers should be multi-homed? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+XOApbZvCIJx1bcRAtkwAJ9vNak3F8FlCf9VDycf6IlAr445nACg59kB w2OWAGdchd2XQyxxgZWQaug= =Yb1+ -END PGP SIGNATURE-
Re: SaidCom disconnected by Level 3 (former Telcove property)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 virendra rode // wrote: Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank Just curious, should small responsive providers should be multi-homed? regards, /virendra - -- Sorry my keyboard came before coffee. I meant to say, shouldn't small responsive providers be multi-homed? regards, /virendra -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+YL3pbZvCIJx1bcRApL+AKCzWJWtAu2L8zVtjUTOMSzELpTMBQCgyomG YdP85Tq6jO77NM5YTsnFQ/Y= =UR8g -END PGP SIGNATURE-
RE: SaidCom disconnected by Level 3 (former Telcove property)
All companies in all industries have a policy of stopping to provide their services if a customer stops paying or violates the contract, I really don't see this as a big/little provider argument. Yes, the small provider should be multi-homed, otherwise a fiber cut or outage can have this same effect. To me the only part of this that is up for argument is did SaidCom actually violate the contract and/or terms of use, and I certainly don't have enough information beyond that one article to make that decision. If someone else does please share with the group. -Scott (As always just my 2 cents) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Frank Bulk Sent: Wednesday, March 14, 2007 10:30 PM To: nanog@merit.edu Subject: SaidCom disconnected by Level 3 (former Telcove property) http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank
Re: SaidCom disconnected by Level 3 (former Telcove property)
Not knowing anything about the case other than what I read in the article, my hang up is that a transit provider can make a phone call and destroy a customer's business with 30 minutes notice. On a DS3 that has actual real lead time to replace, that's a business killer. The argument of should be multi homed holds some water, but I've never considered multihoming as a typical remedy for a 30-90 day outage. And then it only works if lines are underutilized to the point of loosing one will constantly have zero affect on network performance, even during peak use, and if there's still some level of extra redundancy remaining. (Multiple contingency situations aside) My opinion is probably somewhat influenced by the fact that I'm a small ISP with customers that want the internet to NOT be slow, facing that same DS3 lead time problem. I ordered a DS3 in early December, (who's local loop was to ride on a preexisting OC3, sounds easy, right?) and with dates slipping over and over again, and with no firm install date in site provided from the company last week, I had to finally cancel and order with a different company last week. For the last month, the last thing I wanted to do was punt and start the process over again, but at some point, one starts to feel choiceless. Do you think I placed that order in December just for fun? I see talk over and over again on NANOG about Maybe some provider will come in with [insert new technology here] and compete with the cable/DSL providers but as a small provider doing fix broadband wireless, I just don't see how even an army of small providers can compete against the likes of TENS OF BILLIONS OF DOLLARS of cable/telco market capitalization. After fighting Qwest for ten years, maybe I'm starting to feel a little hopeless.
Re: SaidCom disconnected by Level 3 (former Telcove property)
On Wed, 14 Mar 2007, Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Even from that one-sided account, I have serious problems with: Siwert said the Colorado-based Level 3 cited several Internet abuses by SaidCom customers as the reason for the shutdown, including spam problems. Some customers abuse the system, but when that happens, SaidCom contacts the authorities, said Siwert. When we have a customer spamming, we don't call the police. We either talk to, ACL, or shut off the customer. The above suggests to me that SaidCom had spam issues that they were either unable or unwilling to remedy themselves. I also doubt that L3 shut them off without multiple prior warnings, though anything's possible. We had an issue many years ago where a leased line provider (coincidentally borged indirectly into L3) shut off all of our services with no warning at all on a Friday evening. Only after wasting some time with their NOC troubleshooting the circuits were we told we'd been shut off for non-payment. When we eventually got to the bottom of it (many hours later), it turned out they had another customer with a similar name that was way past due...and when billing told their NOC shut off Atlantic they shut off everything that looked like Atlantic even though we were two totally separate and unrelated customers. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: SaidCom disconnected by Level 3 (former Telcove property)
On Mar 15, 2007, at 8:25 PM, Jon Lewis wrote: On Wed, 14 Mar 2007, Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Even from that one-sided account, I have serious problems with: Siwert said the Colorado-based Level 3 cited several Internet abuses by SaidCom customers as the reason for the shutdown, including spam problems. Some customers abuse the system, but when that happens, SaidCom contacts the authorities, said Siwert. When we have a customer spamming, we don't call the police. We either talk to, ACL, or shut off the customer. The above suggests to me that SaidCom had spam issues that they were either unable or unwilling to remedy themselves. ACL's are your friends with non-responsive customers. ;-), But maybe SaidCom did not know better. How many Abuse tickets had they received from TelCove/(3) over what time frame? I may be way off base here: Only knowing the facts presented in the above article, the Abuse complaints may have also included DMCA complaints, which, if not responded to in a timely manner, could also have resulted in liability for (3).* As per the quote above, the abuses included spam, he did not say they were exclusively spam. *I am not a lawyer, nor do I play one on TV. Just reading between the lines here... G
SaidCom disconnected by Level 3 (former Telcove property)
http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? Frank
Re: SaidCom disconnected by Level 3 (former Telcove property)
On Wed, 14 Mar 2007, Frank Bulk wrote: http://www.phillyburbs.com/pb-dyn/articlePrint.cfm?id=1310151 Is this a normal thing for Level 3 to do, cut off small, responsive providers? I heard from a few folks working for TelCove that they were under order to do whatever it took to disconnect customers under certian levels in my local area (not philly). Related? Dunno. Level3 recently stiffed us on a colo contract though, 3 weeks past the no later than install date and we finally canceled it. (mind you that was 7 weeks after the contract was signed). ~shrug~ ..d --- david raistrickhttp://www.netmeister.org/news/learn2quote.html [EMAIL PROTECTED] http://www.expita.com/nomime.html
Re: SaidCom disconnected by Level 3 (former Telcove property)
On 3/14/07, david raistrick [EMAIL PROTECTED] wrote: I heard from a few folks working for TelCove that they were under order to do whatever it took to disconnect customers under certian levels in my local area (not philly). Related? Dunno. Level3 recently stiffed us on a colo contract though, 3 weeks past the no later than install date and we finally canceled it. (mind you that was 7 weeks after the contract was signed). As a former Broadwing customer (now a Level3 customer), things like this worry me a bit... (and tend to keep my up at night). -brandon
Any issues with AS 19548 and their links to Level 3 or TWTC?
Having some connectivity issues with multiple customers on that network from our AS and a few others I've found on traceroute.org; is anyone aware of anything there? Traces in, but which are more likely failing on the return side, often stop at ae-1-0.c1.dfw91.twc-core.net and paix-atl.adelphiacom.net. I've been unsuccessful trying to make them prefer a different inbound route to us. Thanks, David
AboveNet and Level(3) in Chicago
Can anyone confirm if AboveNet and Level(3) have added or improved peering in Chicago? I don't have a previous traceroute to compare to, but a new one from my ISP network singlehomed on AboveNet to a server with several carriers including Level(3) seems to suggest that. Traceroutes always included Level(3), but now they're a few hops shorter and maybe 20 - 30 ms faster. Mike HammettIntelligent Computing Solutionshttp://www.ics-il.com
reminiscing (was re: level 3)
On Nov 11, 2005, at 2:50 PM, [EMAIL PROTECTED] wrote: we clustered the engineers into the IETF terminal room since we're reminiscing, we did this at dallas ietf in 1995, i think it was (yes, http://merit.edu/mail.archives/nanog/2000-11/ msg00222.html). we had hit a timer bug in is-is that was causing routers to lose adjacencies. ravi sat down at a terminal, found the bug in the code, compiled a new image, and we loaded and rebooted the network that evening. nasty, but fun. we don't seem to have fun like this anymore. (maybe it wasn't as simple as ravi finding the bug, but i do seem to remember he fixed this in short order) -b
RE: cogent+ Level(3) are ok now
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Erik Haagsman Sent: Wednesday, November 02, 2005 2:40 PM To: [EMAIL PROTECTED] Cc: John Payne; Patrick W.Gilmore; nanog@merit.edu Subject: Re: cogent+ Level(3) are ok now On Tue, 2005-11-01 at 18:48 -0500, [EMAIL PROTECTED] wrote: On Tue, 01 Nov 2005 11:46:20 EST, John Payne said: What am I missing? Obviously, the same thing that management at SBC is missing: snip He argued that because SBC and others have invested to build high-speed networks, they are due a return. There's going to have to be some mechanism for these people ... to pay for the portion they're using. Why should they be allowed to use my pipes? He offered no details how his idea could be accomplished. For an Internet company to expect to use these pipes free is nuts! Whitacre added for good measure. Quothed --- Well, the other funny thing is that SBC doesn't just spend its own money to build these networks. They get all sorts of help from gov't, etc with taxes and multiple other breaks. I think that was the original complaint. -Drew
RE: cogent+ Level(3) are ok now
On Fri, 4 Nov 2005, Drew Weaver wrote: Well, the other funny thing is that SBC doesn't just spend its own money to build these networks. They get all sorts of help from gov't, etc with taxes and multiple other breaks. I think that was the original complaint. Comcast is wrong, she maintains. It's not like the $30 million subsidy they got to build their corporate headquarters, she said. This crying about subsidies is a little disingenuous. http://www.nytimes.com/2005/10/30/business/yourmoney/30frenzy.html
Re: cogent+ Level(3) are ok now
Richard A Steenbergen wrote: Pete Templin wrote: John Curran wrote: Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s). And there's still revenue, as the traffic is going to customers (we all filter our prefixes carefully, right?). What's the problem with cold-potato again, or should we all just try to double-dip? I can almost smell your sarcasm from here. :) The problem here is that people naively assume all traffic is the same, and costs the same to deliver, which is just not the case. On-net traffic costs significantly more to deliver than outbound traffic, because you are virtually guaranteed that you are going to have to haul it somewhere at your expense. Time out here. John set the stage: cold potato addressed the long haul (or at least that's the assumption in place when I hopped on board). If NetA and NetB are honoring MED (or other appropriate knob), NetA-NetB traffic has already arrived at the closest mutual peering point in the A-B direction. The rest of the infrastructure would be the responsibility of NetB to get the traffic to CustomerPortXYZ, no? How could CustomerXY get any closer to NetA without cutting NetB out of the middle, and if NetB is out of the middle, why should CustomerXY pay NetB anything? pt
Re: cogent+ Level(3) are ok now
On Tue, 2005-11-01 at 18:48 -0500, [EMAIL PROTECTED] wrote: On Tue, 01 Nov 2005 11:46:20 EST, John Payne said: What am I missing? Obviously, the same thing that management at SBC is missing: snip He argued that because SBC and others have invested to build high-speed networks, they are due a return. There's going to have to be some mechanism for these people ... to pay for the portion they're using. Why should they be allowed to use my pipes? He offered no details how his idea could be accomplished. For an Internet company to expect to use these pipes free is nuts! Whitacre added for good measure. /snip Sounds like an extremely short-sighted view of the Net and it's economics. Claiming content providers should be charged for using broadband access-pipes is fine and dandy, but coveniently forgetting that without content there probably wouldn't be a great deal of customers wanting broadband in the first place is a bit sloppy, no? Erik -- Erik Haagsman Network Architect We Dare BV tel: +31.10.7507008 fax: +31.10.7507005 http://www.we-dare.nl
Re: cogent+ Level(3) are ok now
Sounds like an extremely short-sighted view of the Net and it's economics. Claiming content providers should be charged for using broadband access-pipes is fine and dandy, but coveniently forgetting that without content there probably wouldn't be a great deal of customers wanting broadband in the first place is a bit sloppy, no? while valid, this argument plays into the power play game. the key point is that the content providers already paid once for transport [0], as did the content users. we may need more gummint support to keep the rbocs from abusing their subsidized monopoly ownership of the last mile. two years ain't enough to get the cartel-minded out of control of the fcc. randy --- [0] - whether to transit upstreams or by deploying a large network themselves (aol)
Re: cogent+ Level(3) are ok now
On Wed, Nov 02, 2005 at 08:22:20AM -0600, Pete Templin wrote: Time out here. John set the stage: cold potato addressed the long haul (or at least that's the assumption in place when I hopped on board). If NetA and NetB are honoring MED (or other appropriate knob), NetA-NetB traffic has already arrived at the closest mutual peering point in the A-B direction. The rest of the infrastructure would be the responsibility of NetB to get the traffic to CustomerPortXYZ, no? How could CustomerXY get any closer to NetA without cutting NetB out of the middle, and if NetB is out of the middle, why should CustomerXY pay NetB anything? You're forgetting that MEDs suck. When used on real complex production networks, they almost always degrade the quality of the routing. Yes with enough time and energy (or a small enough network) you *can* beat perfect MEDs out of the system (and your customers). You can selectively deaggregate the hell out of your network, then you can zero out all the known aggregate blocks and regions that are in the middle of two MED-speaking interconnection points, and get your customers to tag aggregate blocks announced in multiple locations so that you can zero out those MEDs. With enough time and energy anything is possible, the point is that most folks don't consider it to be worth the time, let alone the customer anger when it degrades your traffic. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: cogent+ Level(3) are ok now
Richard A Steenbergen wrote: Yes with enough time and energy (or a small enough network) you *can* beat perfect MEDs out of the system (and your customers). You can selectively deaggregate the hell out of your network, then you can zero out all the known aggregate blocks and regions that are in the middle of two MED-speaking interconnection points, and get your customers to tag aggregate blocks announced in multiple locations so that you can zero out those MEDs. With enough time and energy anything is possible, the point is that most folks don't consider it to be worth the time, let alone the customer anger when it degrades your traffic. I came up with a reasonably scalable solution using communities and route-map continue, but: CSCsc36517 Externally found severe defect: New (N) Bus Error reload after configure route-map and then clear bgp neighbor Release-note: When a bgp route-map is configured on the router and then clear ip bgp neighbor.. command is executed router experiences Unexpected Reload due to Bus Error. Currently there is no other workaround other than to prevent executing the clear ip bgp neighbor command. ...kinda gets in my way. For what it's worth, it doesn't even require clear ip bgp ne to crash the box. Joy. pt
Re: cogent+ Level(3) are ok now
I don't understand them, either. However, if you define incoming traffic as bad, it encourages depeering by the receiving side if the incoming/outgoing ratio exceeds a certain value, especially among close-to-tier-1 carriers: the traffic does not automatically disappear just because you depeer. Now suppose that the sending side doesn't want to play games and buys transit from one of your other peers. Given the tier-1 status, there is some chance that this has a measurable impact on the traffic ratio with that other peer. Essentially, this is a self-fulfilling prophecy, and it works equally well if you define outgoing traffic as bad. I was trying to stay out of this. But I think I'm going to chime in here. I think (originally)... means... whenever the NAP structure was created and the NSFnet was decommissioned, long haul moving of the bits was very expensive. Perhaps more so than the local-distance piece because you had comparatively few of these links and had to cart bits a very long way to dump them off. Now I believe that with the influx of large, reasonable colos and 20+ high speed interconnects in a region (or slightly larger area). This coupled with dramatically reduced long haul costs has shifted the value/expense ratios in peering. For example if you are a content network. If you needed to peer in 5 places in the old (long-haul=expensive) model would colo your content in 5 places near your interconnected and the only interconnections between your colos would be whatever you needed to keep heartbeat and data sync between them. But you incurred a large cost for this colo and manpower and other things. Now... let's just say you don't need to do that... because you can run a few circuits around the country (or MPLS them) and its pretty cheap. However, the last-mile piece for access networks HASN'T moved as much in the same time period. Lots of networks (Q, LVLT, GBLX, etc) built long haul networks. Lots of them colo'd in ILEC switch (and tandem buildings). But this doesn't do ISP type businesses very much good. You still have to pay interconnect fees to the ILEC or exorbitant colo fees to them to backhaul the circuit to your DC with all of your equipment. Means... that even though you control the customer and the CPE, you are paying fees to many folks that _really_ own the copper (or coax) or underlying infrastructure and they have little to NO competitive pressure to be more than slightly price concerned. This drives you to the idea that actually moving higher PPS is bad while increasing peak speeds is good. Customers are buying/being marketed to by peak speed. So you give them large pipes because once you've paid the underlying tariffs to get to their house/business (say a dry pair) whatever speed you signal on it is your equipment cost, nothing else. But actually encouraging them to USE that bandwidth is expensive because your cost of growing the T3, OC3, OC12 or whatever you are backhauling from the various COs to your network is BAD and expensive. This is the model as I understand it today. Formerly the longhaul and cost of transit was so expensive these costs were more negligible. Now we have a playing field. Now if you depeer a guy [Cogent] for example, and you force him to buy transit from someone who doesn't have a very vibrant transit business [NTT/Verio, I'm being kind] you increase his costs and force [NTT/Verio] to upgrade their network which may take time. The depeered guy suffers. Maybe he doesn't suffer much at all [Like when Cogent could force all of its AOL traffic through its Level3 connection]. But his customers... They want speedy access to your eyeballs. Maybe some of them will want to reduce the number of networks traversed to get to your eyeballs. By limiting the number of SFI connections you have... I would theorize you can force those who can afford it to interconnect with you directly. Not everyone. But a few. If you perceive your network as better than your SFI peers, then naturally you would assume that the business of their customers would eventually flow to you. The problem with this kind of increased instability is that the perception and tenacity of all the players is very difficult to assess and is often not as vastly different as the players would hope. But to the extent you can force others to have to redo their network interconnections with little impact [at least what you believe when you are taking the action] the better it is for you. Especially if you are running out of value-added sales pitches. Further proving the counterpoint: Large eyeball networks (like Verizon broadband) that use Level3 (formerly genuity) a lot, didn't get affected by this depeering. Why? Because they've already added diversity to their network. Even if the Level3 routes are normally chosen more often than the other providers [guessing, not saying its true], Level3 forced their 95th percentile to peak with
Re: cogent+ Level(3) are ok now
On Wed, Nov 02, 2005 at 02:44:20PM -0600, Pete Templin wrote: I came up with a reasonably scalable solution using communities and route-map continue, but: For what value of scalable? --Jeff
Re: cogent+ Level(3) are ok now
On Wed, 2 Nov 2005, Jeff Aitken wrote: On Wed, Nov 02, 2005 at 02:44:20PM -0600, Pete Templin wrote: I came up with a reasonably scalable solution using communities and route-map continue, but: For what value of scalable? anything, its 'scalable' :) Steve
Re: cogent+ Level(3) are ok now
Jeff Aitken wrote: On Wed, Nov 02, 2005 at 02:44:20PM -0600, Pete Templin wrote: I came up with a reasonably scalable solution using communities and route-map continue, but: For what value of scalable? For me, plenty, but a four-POP single-state network usually has different constraints on scalable. However, I'd categorize it as one community-list per MED tier (i.e. if you just want near/far, that's two tiers, etc.) and one community-list entry per POP (or group of POPs, if you have some grouping logic embedded in your internal communities). pt
Re: cogent+ Level(3) are ok now
For me, plenty, but a four-POP single-state network usually has different constraints on scalable. However, I'd categorize it as one community-list per MED tier (i.e. if you just want near/far, that's two tiers, etc.) and one community-list entry per POP (or group of POPs, if you have some grouping logic embedded in your internal communities). I think Pete is saying that as long as you aren't a control-nazi, its a good system. :) Given that constraint, I wonder how of NANOG it applies to. ;) Deepak
Re: cogent+ Level(3) are ok now
On Wed, Nov 02, 2005 at 05:13:27PM -0600, Pete Templin wrote: For me, plenty, but a four-POP single-state network usually has different constraints on scalable. Right. On Wed, Nov 02, 2005 at 06:20:39PM -0500, Deepak Jain wrote: I think Pete is saying that as long as you aren't a control-nazi, its a good system. :) My point wasn't that his system doesn't work for him; presumably it does. My point was that in the context of global peering, what works for a small network may not (and probably does not) work for someone the size of, say, Level3. There are a lot of operational issues that arise if you listen to MEDs from peers. Apart from the minor issues like oscillations, ratchet-down, and packing inefficiencies, there is the fundamental problem that you will see potentially significant churn as a result of changes in your peers' networks. There is also the problem that there is no single best exit point for many large prefixes (e.g., if you peer with L3, there is no one best place to send traffic destined for something inside 4/8). --Jeff
Re: cogent+ Level(3) are ok now
Hi John, Even with cold-potato routing, there is an expense in handling increased levels of traffic that is destined for your network. This increase in traffic often has no new revenue associated with it, because it is fanning out to thousands of flat-rate consumer/small-business connections (e.g. DSL) where billing is generally by peak capacity not usage. not true for cogent tho, we know that virtually all their traffic is usage based transit customers Steve
Re: cogent+ Level(3) are ok now
At 12:27 PM + 11/1/05, Stephen J. Wilcox wrote: Hi John, Even with cold-potato routing, there is an expense in handling increased levels of traffic that is destined for your network. This increase in traffic often has no new revenue associated with it, because it is fanning out to thousands of flat-rate consumer/small-business connections (e.g. DSL) where billing is generally by peak capacity not usage. not true for cogent tho, we know that virtually all their traffic is usage based transit customers The traffic from Cogent creates additional infrastructure requirements on L3. L3 may (or may not) be able to recover these costs as incremental revenue from the recipients, depending on the particulars of their agreements. One way of mitigating their exposure is to set an upper bound on uncompensated inbound traffic. Mind you, this is entirely hypothetical, as specifics of the Cogent/L3 agreement are not available. However, it is one way to let everyone bill and keep for Internet traffic without an unlimited exposure, and it is an approach that has been used successfully in the past. /John
Re: cogent+ Level(3) are ok now
On Nov 1, 2005, at 7:53 AM, John Curran wrote: At 12:27 PM + 11/1/05, Stephen J. Wilcox wrote: Hi John, Even with cold-potato routing, there is an expense in handling increased levels of traffic that is destined for your network. This increase in traffic often has no new revenue associated with it, because it is fanning out to thousands of flat-rate consumer/small-business connections (e.g. DSL) where billing is generally by peak capacity not usage. not true for cogent tho, we know that virtually all their traffic is usage based transit customers The traffic from Cogent creates additional infrastructure requirements on L3. L3 may (or may not) be able to recover these costs as incremental revenue from the recipients, depending on the particulars of their agreements. One way of mitigating their exposure is to set an upper bound on uncompensated inbound traffic. Mind you, this is entirely hypothetical, as specifics of the Cogent/ L3 agreement are not available. However, it is one way to let everyone bill and keep for Internet traffic without an unlimited exposure, and it is an approach that has been used successfully in the past. Taking L3 Cogent completely out of this discussion, I'm not sure I agree with your assessment. I think everyone agrees that unbalanced ratios can create a situation where one side pays more than the other. However, assuming something can be used to keep the costs equal (e.g. cold-potato?), I do not see how one network can tell another: You can't send me what my customers are requesting of you. If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. -- TTFN, patrick
Re: cogent+ Level(3) are ok now
At 9:40 AM -0500 11/1/05, Patrick W. Gilmore wrote: I think everyone agrees that unbalanced ratios can create a situation where one side pays more than the other. However, assuming something can be used to keep the costs equal (e.g. cold-potato?), Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s). I do not see how one network can tell another: You can't send me what my customers are requesting of you. Depeering seems to say exactly that, no? If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. Agreed... I'm not defending the business model, only pointing out that some folks may find it easier to bill their peers than customers. /John
Re: cogent+ Level(3) are ok now
John Curran wrote: Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s). And there's still revenue, as the traffic is going to customers (we all filter our prefixes carefully, right?). What's the problem with cold-potato again, or should we all just try to double-dip? pt
Re: cogent+ Level(3) are ok now
On Tue, 1 Nov 2005, John Curran wrote: I do not see how one network can tell another: You can't send me what my customers are requesting of you. Depeering seems to say exactly that, no? No. Presumably, that traffic's still going to be exchanged between the two networks' customers. Depeering just says go pay someone for transit if you want to talk to our network. Not talking to a network that depeers you is not a long term viable option if you're in the internet access provider business. If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. Agreed... I'm not defending the business model, only pointing out that some folks may find it easier to bill their peers than customers. Seems like some people want to bill both. Not being an expert in Tier1 peering issues, it really seems like the only explanation for this depeering was L3 wanting to raise Cogent's cost of doing business...presumably as an attack on Cogent's business model of selling access way below the average Tier1 going rate. For those who disagree, how does forcing Cogent to pay [anyone, not necessarily L3] for access to L3's network reduce L3's cost of carrying the bits that will flow regardless of whether Cogent's peering with L3 or buying transit to get to L3? I actually can think of a couple possible explanations. Perhaps L3 wanted Cogent to interconnect with them in more places (so they wouldn't have to carry traffic as far), and Cogent refused. If you have a customer in CA, and I have a customer in FL, and we peer, whats a fair way to move that traffic cross country? i.e. We both bill our customers...who pays to move the bits cross country? -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: cogent+ Level(3) are ok now
Pete Templin wrote: John Curran wrote: Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s). And there's still revenue, as the traffic is going to customers (we all filter our prefixes carefully, right?). What's the problem with cold-potato again, or should we all just try to double-dip? pt ah yes, double dipping. On-net traffic should be charged a lot less, because after all, it is double dipping. /vijay
Re: cogent+ Level(3) are ok now
On Nov 1, 2005, at 9:40 AM, Patrick W. Gilmore wrote: If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. That is something that has always confused me about ratio based peering disputes. Surely it is the responsibility of the content-sucking network to build and engineer to meet the demands of *their* customers (and build the cost of doing that into the pricing model). It appears to me that the content heavy networks are going above and beyond to work around the broken model that the content-suckers have. What am I missing?
Re: cogent+ Level(3) are ok now
On Tue, 1 Nov 2005, John Payne wrote: What am I missing? That it's a pure power play. Peering is only distantly associated with costs or responsibilities. It has to do with what company has the intestinal fortitude to draw a line in the sand and stick with it no matter how many customers cancel their service. Those with a critical mass of traffic and the right amount of guts win. Everyone else loses the peering game. -- Brandon Ross AIM: BrandonNRoss Director, Network Engineering ICQ: 2269442 Internap Skype: brandonross Yahoo: BrandonNRoss
Re: cogent+ Level(3) are ok now
On Tue, 1 Nov 2005, Brandon Ross wrote: On Tue, 1 Nov 2005, John Payne wrote: What am I missing? That it's a pure power play. market position is important Peering is only distantly associated with costs or responsibilities. no, peering is entirely associated with costs or responsibilities.. what other reason is there to peer ? It has to do with what company has the intestinal fortitude to draw a line in the sand and stick with it no matter how many customers cancel their service. have to weigh up the gains and losses to see if that is a good or bad thing tho. Those with a critical mass of traffic and the right amount of guts win. markets are always stacked in favour of the larger players in that way.. saying 'hey i'm a little guy, give me chance' generally goes unheard Everyone else loses the peering game. not peering isnt necessarily losing, there are networks who would peer with me if i turned up in asia or the west coast, but my cost to get there is greater than sticking to transit. to get a new peer, both sides need to feel they are gaining value Steve
Re: cogent+ Level(3) are ok now
On Nov 1, 2005, at 11:46 AM, John Payne wrote: On Nov 1, 2005, at 9:40 AM, Patrick W. Gilmore wrote: If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. That is something that has always confused me about ratio based peering disputes. Surely it is the responsibility of the content-sucking network to build and engineer to meet the demands of *their* customers (and build the cost of doing that into the pricing model). It appears to me that the content heavy networks are going above and beyond to work around the broken model that the content-suckers have. What am I missing? That argument works in both directions. I'm an eyeball network, I'll sit in my DLSAM and force all the content people to come to me. Isn't their responsibility to their customers to deliver bits to me? Assume that both content and eyeballs are equally important. (If you assume one is more important than the other, this all devolves into the less important should pay, period, which is not going to happen.) Why does the content network get to dump traffic instantly without paying for long haul, but the eyeballs have to carry it across the ocean / country / whatever? You could argue that's The Way It Is. Eyeball and Tier One networks appear to disagree. Not sure they are wrong. It seems reasonable (to me, at least) to ask that a peer share the cost of trading bits. Cold-potato does not mean the content network has to deliver bits to every DSLAM in the country. But asking the hosting provider with 10M ft^2 colos in SJC IAD to carry some of that traffic to ORD, DFW, LAX, JFK, etc., seems like a fair compromise. -- TTFN, patrick
Re: cogent+ Level(3) are ok now
On Nov 1, 2005, at 10:04 AM, John Curran wrote: At 9:40 AM -0500 11/1/05, Patrick W. Gilmore wrote: I think everyone agrees that unbalanced ratios can create a situation where one side pays more than the other. However, assuming something can be used to keep the costs equal (e.g. cold- potato?), Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s). Which is COMPLETELY AND TOTALLY irrelevant to the peer network. If your network can't cover the cost of delivering bits from the DSLAM to the CPE, why in the hell are you in this business? You've been doing this for a very, very long time John. I know you know better. Stop trying to confuse the newbies. I do not see how one network can tell another: You can't send me what my customers are requesting of you. Depeering seems to say exactly that, no? Only if you are Cogent / L3 (or Cogent / FT, or Cogent / Teleglobe, or Cogent / $NEXT-DEPEER). Any other time a network gets de-peered, the bits still flow. So I repeat, how can an eyeball network tell a content provider: You can't send me what my customers are requesting of you. The only way I can think to do that is to intentionally congest the path. (Which many eyeball networks actually do, now that I think about it.) But that might have an adverse affect on your customer growth. If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. Agreed... I'm not defending the business model, only pointing out that some folks may find it easier to bill their peers than customers. I doubt they will succeed - at least in the long run, or even in the majority of cases. But stranger things have happened. Just remember, turn-about it fair play. So they should be careful what they wish for. -- TTFN, patrick
Re: cogent+ Level(3) are ok now
On Tue, 1 Nov 2005, Stephen J. Wilcox wrote: On Tue, 1 Nov 2005, Brandon Ross wrote: On Tue, 1 Nov 2005, John Payne wrote: What am I missing? That it's a pure power play. market position is important If by market position you are referring to who needs/wants/can do without the traffic more, yes. Peering is only distantly associated with costs or responsibilities. no, peering is entirely associated with costs or responsibilities.. what other reason is there to peer ? I was probably being a bit too dramatic with that statement. What I'm trying to get across is that it doesn't matter who is supposed to pay for their customers' traffic. It doesn't matter that I have a million dialup users, if I can use my market position to get someone else to peer with me for free that's all that matters. The fact that those 1 million customers pay me is irrelevant. It has to do with what company has the intestinal fortitude to draw a line in the sand and stick with it no matter how many customers cancel their service. have to weigh up the gains and losses to see if that is a good or bad thing tho. Of course. Those with a critical mass of traffic and the right amount of guts win. markets are always stacked in favour of the larger players in that way.. saying 'hey i'm a little guy, give me chance' generally goes unheard Quite true. Everyone else loses the peering game. not peering isnt necessarily losing, there are networks who would peer with me if i turned up in asia or the west coast, but my cost to get there is greater than sticking to transit. You don't have to tell me that, I work for Internap, we've made a business out of not peering, and doing quite well at it. I said loses the peering game. I didn't say they lost the game in entirety. Similarly, just because a company wins the peering game (fully peered with all other default free networks) doesn't mean it wins the business game. Just take a look at a former employer of mine, 4006 was default free, but that doesn't mean that we made any money. to get a new peer, both sides need to feel they are gaining value Or one side needs to be more scared of the other side cutting them off. -- Brandon Ross AIM: BrandonNRoss Director, Network Engineering ICQ: 2269442 Internap Skype: brandonross Yahoo: BrandonNRoss
Re: cogent+ Level(3) are ok now
On Tue, Nov 01, 2005 at 11:16:58AM -0500, vijay gill wrote: Pete Templin wrote: John Curran wrote: Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s). And there's still revenue, as the traffic is going to customers (we all filter our prefixes carefully, right?). What's the problem with cold-potato again, or should we all just try to double-dip? pt ah yes, double dipping. On-net traffic should be charged a lot less, because after all, it is double dipping. I can almost smell your sarcasm from here. :) The problem here is that people naively assume all traffic is the same, and costs the same to deliver, which is just not the case. On-net traffic costs significantly more to deliver than outbound traffic, because you are virtually guaranteed that you are going to have to haul it somewhere at your expense. People expect their sub $10/Mbps transit pricing for all services across the board now, without understanding that those rates are ONLY sustainable because of negligible longhaul costs for the outbound traffic. On-net traffic is not double dipping, it is the ony way that transit can be sold for a particular price. So does that mean that anyone with outbound heavy traffic is automatically taking advantage of a peer? Of course not, because while some types of traffic may indeed cost more to deliver, that traffic is usually *gasp* billed at a higher rate too. Other than spot markets like Cogent trying to prop up its ratios or a small tier 2/3 taking advantage of a 95th percentile billing trick to give away free inbound, I would challange folks to find ordinary markets where inbound traffic is not priced substantially higher than outbound, especially in areas outside of the big tier 1 bandwidth cities. Numbers close to $100/Mbps (or higher) are still perfectly common on OC3's, even on cities which are on major longhaul fiber routes. Remember that content can be moved in order to reduce the cost, eyeballs can not. CDN's deliver bits to the right areas to bypass transport costs, and even ordinary folks choose where to install their servers in order to maximize quality and lower price. Content people who buy transit routinely put their servers at or near major ix facilities in order to get a lower price for the traffic (hey look my content goes in and out the same pop, or even the same router). Yes there is an associated cost to deliver access traffic to far-flung regions, but your customers are paying you a higher rate to do it too. So, what is inherently wrong with content customers paying $10/Mbps for a service which is substantially cheaper to provide, and the access customers paying $70/Mbps for the same thing? A lot of people seem to be taking the position of silent resentment towards the folks who are selling content heavy bandwidth at what can only be described as competetive market pricing (meaning, you can buy it at that price from almost anyone). They see such a large volume of traffic and think: a) crap, our network design can't possibly deliver that many bits at those prices in order to compete with them. and b) but man if we were billing all that at $70/Mbps we could, and we would be if not for that damn content-heavy network who is getting free peering in to our network in order to sell it for so cheap. We're paying more of the cost for that traffic than they are too, clearly we need to depeer them. Unfortunately they often do so without understanding the symbiotic relationship between the two kinds of traffic, and the two types of networks. If you look at a network like Cogent, it is designed from the ground up to be efficient and cheap at delivering bulk bits from a few customers at a few key points to the rest of the Internet, which is how Cogent is able to erm lose as little money as they do. Their network design looks almost nothing like a network who is optimized to deliver access circuits to a large number of smaller customers across a large number of locations, and it would be far less efficient at it if called upon to do so. In this case, jealousy is blinding a lot of people to the fact that there is room for networks who specialize in content to co-exist with networks who specialize in access, and for them both to add value to each other through interconnection. Specializing in a specific area leads to optimized network designs and reduced costs, and networks who don't may find that they aren't very good (or at least, cost competetive) at either. This naturally leads into two camps: 1) Networks who are more efficient, who end up paying a lot less, and who end up moving a very large amount of bits because of it (but at a much lower price/meg). 2) Networks who are less efficient, who pay a lot more, and who therefore have to charge their customers a lot more in order to survive. These
Re: cogent+ Level(3) are ok now
On Tue, 01 Nov 2005 11:46:20 EST, John Payne said: That is something that has always confused me about ratio based peering disputes. Surely it is the responsibility of the content-sucking network to build and engineer to meet the demands of *their* customers (and build the cost of doing that into the pricing model). It appears to me that the content heavy networks are going above and beyond to work around the broken model that the content-suckers have. What am I missing? Obviously, the same thing that management at SBC is missing: http://www.marketwatch.com/news/story.asp?guid=%7B5A606A5A%2D18D7%2D4FC9%2DA65C%2DC7317BC7E1CB%7Damp;siteid=mktw WASHINGTON (MarketWatch) --- The chief executive of SBC Communications Inc. thinks companies doing business on the Internet, such as Microsoft Corp. and Vonage Inc., are due for a wake-up call. How do you think they're going to get to customers? Through a broadband pipe. Cable companies have them. We have them, said Ed Whitacre in a BusinessWeek Online interview. What they would like to do is use my pipes for free. I ain't going to let them do that. He argued that because SBC and others have invested to build high-speed networks, they are due a return. There's going to have to be some mechanism for these people ... to pay for the portion they're using. Why should they be allowed to use my pipes? He offered no details how his idea could be accomplished. For an Internet company to expect to use these pipes free is nuts! Whitacre added for good measure. pgpzHLrNq12mC.pgp Description: PGP signature
Re: cogent+ Level(3) are ok now
for a totally different spin, my little router mess (not daytime job) is starting to depeer folk who intentionally deaggregate. and gosh, my config builds sure run faster! randy --- From: Randy Bush [EMAIL PROTECTED] Date: Tue, 1 Nov 2005 16:22:43 -1000 To: [EMAIL PROTECTED] Subject: six depeering 3130 plans do drop peering with at the seattle ix at 23:59 on 2005.11.02, i.e. about 22 hours from now. the reason is that your deaggregation is a detriment to the net. randy
Re: cogent+ Level(3) are ok now
At 12:56 AM -0400 10/29/05, Daniel Golding wrote: I have no specific information, but I'm guessing there is a per-mbps charge that kicks in at certain ratio levels. ... I'm having a bit of trouble figuring out Level(3)'s goal in all this. A bit of incremental revenue? For all of this trouble? I could understand feeling that Cogent's ratios are a violation of their peering requirements and depeering them on principle, but if that's the case, why back down for a little cash? I do not have any information on this particular arrangement, but can speak to one possibility... Even with cold-potato routing, there is an expense in handling increased levels of traffic that is destined for your network. This increase in traffic often has no new revenue associated with it, because it is fanning out to thousands of flat-rate consumer/small-business connections (e.g. DSL) where billing is generally by peak capacity not usage. It's also true that some of the most popular Internet destinations will receive transit at bargain rates because of their relative size and buying power. A settlement fee that kicks in only on egregious ratios allows one to more freely interconnect without bearing the full cost burden should the traffic become wildly asymmetric. /John
cogent+ Level(3) are ok now
http://biz.yahoo.com/prnews/051028/laf022.html?.v=27 The internet will not end on November(9)th :) - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
RE: cogent+ Level(3) are ok now
Now, one really needs to wonder why the agreement could not be reached *prior* to the depeering on 10/5 It's not rocket science. It's only as complex as one makes it out to be. (one can attempt to explain away the complexities, but they apparently were able to *finalize* an agreement in 3 weeks, perhaps the agreement happened in it's entirety in 3 weeks - no speculation on the agreement is required unless you have nothing better to do) Who are the next discontent couples? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jared Mauch Sent: Friday, October 28, 2005 11:08 AM To: nanog@merit.edu Subject: cogent+ Level(3) are ok now http://biz.yahoo.com/prnews/051028/laf022.html?.v=27 The internet will not end on November(9)th :) - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine. -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.362 / Virus Database: 267.12.5/150 - Release Date: 10/27/2005
Re: cogent+ Level(3) are ok now
...the companies have agreed to the settlement-free exchange of traffic subject to specific payments if certain obligations are not met. So it does look like Cogent bent somwhat...I'm guessing they agreed to pay some sort of traffic imbalance fee? Anyone know of any other peering arrangements that have similar terms? I'll admit, that's a new one for me... -C On Oct 28, 2005, at 2:31 PM, Eric Louie wrote: Now, one really needs to wonder why the agreement could not be reached *prior* to the depeering on 10/5 It's not rocket science. It's only as complex as one makes it out to be. (one can attempt to explain away the complexities, but they apparently were able to *finalize* an agreement in 3 weeks, perhaps the agreement happened in it's entirety in 3 weeks - no speculation on the agreement is required unless you have nothing better to do) Who are the next discontent couples? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jared Mauch Sent: Friday, October 28, 2005 11:08 AM To: nanog@merit.edu Subject: cogent+ Level(3) are ok now http://biz.yahoo.com/prnews/051028/laf022.html?.v=27 The internet will not end on November(9)th :) - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine. -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.362 / Virus Database: 267.12.5/150 - Release Date: 10/27/2005
Re: cogent+ Level(3) are ok now
Christopher Woodfield wrote: ...the companies have agreed to the settlement-free exchange of traffic subject to specific payments if certain obligations are not met. So it does look like Cogent bent somwhat...I'm guessing they agreed to pay some sort of traffic imbalance fee? There are other possibilities. Maybe they agreed to pay a transit fee should they fail to carry the L3 user's requested traffic as far as possible before handing it off (cold potato routing) and hand it off at the earliest possibility (hot potato routing) leaving L3 to backhaul it across the L3 network to the user who requested the data. Etc. jc
Re: cogent+ Level(3) are ok now
Eric Louie wrote: Now, one really needs to wonder why the agreement could not be reached *prior* to the depeering on 10/5 It's not rocket science. As people have pointed out repeatedly, this was surely not rocket science since it wasn't a technical problem at all. It was a business conflict. It seems clear to me what probably happened. First-round negotaitions failed 'cause Level 3 thought Cogent was bluffing (and perhaps vice versa). Level 3 called the bluff, but it wasn't a bluff, and Level 3 then blinked (or so it appears from reading between the lines of what I've seen). They both got back to negotiation, and with a better understanding of to how much pain the other willing to take to get what they want, this time they came out with an agreement. Doesn't seems mysterious. [snip] Who are the next discontent couples? And how do I protect myself and my customers from any problems these kinds of events cause regardless of who the next players might be? -- Crist J. Clark [EMAIL PROTECTED] Globalstar Communications(408) 933-4387
Re: cogent+ Level(3) are ok now
On 10/28/05 5:45 PM, JC Dill [EMAIL PROTECTED] wrote: Christopher Woodfield wrote: ...the companies have agreed to the settlement-free exchange of traffic subject to specific payments if certain obligations are not met. So it does look like Cogent bent somwhat...I'm guessing they agreed to pay some sort of traffic imbalance fee? There are other possibilities. Maybe they agreed to pay a transit fee should they fail to carry the L3 user's requested traffic as far as possible before handing it off (cold potato routing) and hand it off at the earliest possibility (hot potato routing) leaving L3 to backhaul it across the L3 network to the user who requested the data. I doubt it. Cold potato is normally the first thing Cogent offers in a situation like this. I'm guessing this went something beyond that. Cogent would have offered cold potato well before the original depeering. I have no specific information, but I'm guessing there is a per-mbps charge that kicks in at certain ratio levels. Or, there may be a flat port charge per month under certain conditions - Sprint did this many years ago. Etc. jc I'm having a bit of trouble figuring out Level(3)'s goal in all this. A bit of incremental revenue? For all of this trouble? I could understand feeling that Cogent's ratios are a violation of their peering requirements and depeering them on principle, but if that's the case, why back down for a little cash? Of course, various external pressures may have been brought to bear on Level(3). Customers, regulators, press, creditors, etc. - Dan
Re: cogent+ Level(3) are ok now
On 10/28/05 7:37 PM, Crist Clark [EMAIL PROTECTED] wrote: Eric Louie wrote: Now, one really needs to wonder why the agreement could not be reached *prior* to the depeering on 10/5 It's not rocket science. As people have pointed out repeatedly, this was surely not rocket science since it wasn't a technical problem at all. It was a business conflict. It seems clear to me what probably happened. First-round negotaitions failed 'cause Level 3 thought Cogent was bluffing (and perhaps vice versa). Level 3 called the bluff, but it wasn't a bluff, and Level 3 then blinked (or so it appears from reading between the lines of what I've seen). They both got back to negotiation, and with a better understanding of to how much pain the other willing to take to get what they want, this time they came out with an agreement. Doesn't seems mysterious. It should. Level(3) knew that Cogent would partition. Why? Because they've done it before, more than once. Their business model supports that strategy (some would say, demands it). The Level(3) folks are well informed and would certainly have anticipated this action. The Cogent folks also knew, with a high degree of probability, that Level(3) would carry out their threat. No one sends out a depeering letter unless they are willing to pull the plug. Why? Because sometimes the other party pre-empts you and downs the session before you can. Peering is one of those things that seems very simple. On the small scale that is correct. On the larger scale, especially when dealing with SFI networks, the rules change and things get hairy. Things like ratios matter a great deal when your traffic is in a zero-sum condition with ratio sensitive SFI peers. Cogent is an interesting case, as their peering decisions are typically made with more-than-ordinarily ruthlessness. [snip] - Dan
Re: Level 3 RFO
* Daniel Roesen: On Sun, Oct 23, 2005 at 09:48:58PM +0200, Florian Weimer wrote: This isn't the first time this has happened to an ISP. 8-( Indeed. Are there any configuration tweaks which can locally confine such an event? Something like the hard prefix limit for BGP, perhaps. JunOS: set protocols ospf prefix-export-limit n set protocols isis level n prefix-export-limit n Wouldn't an import limit be better? If you've got a almost-fully-meshed MPLS core, export limits won't really work, will they? In more traditional networks, I can imagine that it helps to confine anomalies. Has anybody tried that on a real network? 8-)
Re: Level 3 RFO
On Mon, Oct 24, 2005 at 01:25:23PM +0200, Florian Weimer wrote: Are there any configuration tweaks which can locally confine such an event? Something like the hard prefix limit for BGP, perhaps. JunOS: set protocols ospf prefix-export-limit n set protocols isis level n prefix-export-limit n Wouldn't an import limit be better? We're talking link-state protocols here... they need to have the same view everywhere. The only thing you can limit is what you inject into the (IGP-)global view. If you've got a almost-fully-meshed MPLS core, export limits won't really work, will they? I don't understand this question. What has MPLS to do with IGP route filtering?!? Regards, Daniel -- CLUE-RIPE -- Jabber: [EMAIL PROTECTED] -- [EMAIL PROTECTED] -- PGP: 0xA85C8AA0
Re: Level 3 RFO
* Daniel Roesen: On Mon, Oct 24, 2005 at 01:25:23PM +0200, Florian Weimer wrote: Are there any configuration tweaks which can locally confine such an event? Something like the hard prefix limit for BGP, perhaps. JunOS: set protocols ospf prefix-export-limit n set protocols isis level n prefix-export-limit n Wouldn't an import limit be better? We're talking link-state protocols here... they need to have the same view everywhere. The only thing you can limit is what you inject into the (IGP-)global view. What a pity. There isn't an ugly workaround, either? There has to be something that can be done, given the operational risk that is involved. Certainly, this adds a new dimension to the distributed single point of failure concept. 8-( If you've got a almost-fully-meshed MPLS core, export limits won't really work, will they? I don't understand this question. What has MPLS to do with IGP route filtering?!? It's the almost fully-meshed part. In such a setup, a single router which exceeds the limit can affect a large part of the the network, even if other routers do not propagate the bogus data. But as you say, if the limit you mentioned is just a local limit on redistribution to the IGP for a single router, my point is moot--if it's in the IGP, you lose because the limit does not apply to routes which are received over the IGP.
Re: Level 3 RFO
However, due to the number of flooded LSAs, other devices in the Level 3 network had difficulty fully loading the OSPF tables and processing the volume of updates. This caused abnormal conditions within portions of the Level 3 network. Manual intervention on specific routers was required to allow a number of routers to return to a normal routing state. This isn't the first time this has happened to an ISP. 8-( Are there any configuration tweaks which can locally confine such an event? Something like the hard prefix limit for BGP, perhaps. (I'm not an OSPF expert, and understand that things are generally more difficult with link-state protocols.)
Re: Level 3 RFO
On Sun, Oct 23, 2005 at 09:48:58PM +0200, Florian Weimer wrote: This isn't the first time this has happened to an ISP. 8-( Indeed. Are there any configuration tweaks which can locally confine such an event? Something like the hard prefix limit for BGP, perhaps. JunOS: set protocols ospf prefix-export-limit n set protocols isis level n prefix-export-limit n I'm told IOS has the ~same. Best regards, Daniel -- CLUE-RIPE -- Jabber: [EMAIL PROTECTED] -- [EMAIL PROTECTED] -- PGP: 0xA85C8AA0
Level 3 RFO
Customer Information Customer Company Name: (Internap) Customer Contact Information: ([EMAIL PROTECTED]) Customer Location: (All services with Level3 Communications) Original Ticket Number: SM Parent 1429209 Customer Impact: Outage Event Summary Outage location: IP North America, Trans-Atlantic and European Markets Ticket Create Date and Time: 10/21/2005 12:01 MDT Service Restore Date and Time: Between 10/21/2005 12:25 MDT to 10/21/2005 5:31 MDT depending on Location Total Duration: Varied by Location Event Description: A configuration update was applied to an edge router in Chicago as part of approved low risk maintenance activity. This validated and approved configuration change was applied to four other major markets with no impact. However; in this specific case the configuration was corrupted during the deployment process on this specific edge router. Upon load of the corrupted configuration, the device created an open-ended policy allowing this router's routes to be redistributed to OSPF. The engineering team immediately reverted to the previous saved configuration to mitigate route propagation. The rollback was followed by deliberate router isolation and complete device reload to ensure no stale LSAs (Link State Announcements), existed on the device and completed by 12:08 MDT. After reloading the edge router, the initial cause of the event was effectively mitigated. However, due to the number of flooded LSAs, other devices in the Level 3 network had difficulty fully loading the OSPF tables and processing the volume of updates. This caused abnormal conditions within portions of the Level 3 network. Manual intervention on specific routers was required to allow a number of routers to return to a normal routing state. Root Cause Analysis Committed redistribution of loopback statement in an erroneous state. Repair On devices with large number of adjacent neighbors a selective process of disabling interfaces on redundant paths or OSPF process restarts stabilized the affected portions to the network. Future Preventive Actions The Level 3 engineering team is currently analyzing the event in order to determine an appropriate action plan. Details of this specific plan will be available after the analysis is complete.
Re: Level 3's side of the story
On 16.10 16:04, Simon Leinen wrote: Kevin Loch writes: Does anyone have reachability data for c-root during this episode? The RIPE NCC DNSMON service has some: http://dnsmon.ripe.net/dns-servmon/server/plot?server=c.root-servers.nettype=dropststart=1128246543tstop=1128972253 If there is anyone with more comprehensive data I'd like to hear about it ;-) ! According to BGPlay for that particular prefix from Route-Views data (for some reason the RIPE RIS server used by BGPlay seems to be down at the moment), the episode seems to be between these times (UTC): It is up now. Of the two routes to c.root-servers.net via Level 3 in that data set one stayed up during the period and the other got healed immediateely via AS286. It flipped back later. ... The interval in the URL above starts 72 hours before the start of the episode and ends 72 hours after its end. I cannot see any particular problems that would coincide with the episode, from that set of probes (RIPE TTM). I agree that there is nothing really significant here. It is mostly noise. What one is looking for in these graphs is strong vertical patterns. dnsmon did not detect anything significant concerning reachability of c.root-servers.net. As someone else said, partial unreachability of a particular root nameserver isn't that much of an issue. Indeed. But it's an interesting question nevertheless. A red instance of a fish from the northern seas ? Daniel
Re: Level 3's side of the story
Kevin Loch writes: Does anyone have reachability data for c-root during this episode? The RIPE NCC DNSMON service has some: http://dnsmon.ripe.net/dns-servmon/server/plot?server=c.root-servers.nettype=dropststart=1128246543tstop=1128972253 According to BGPlay for that particular prefix from Route-Views data (for some reason the RIPE RIS server used by BGPlay seems to be down at the moment), the episode seems to be between these times (UTC): 2005-10-05 09:49:03 Route Withdrawal ( 3356 174 2149 ) 2005-10-07 19:24:13 Route Announcement 3356 174 2149 The interval in the URL above starts 72 hours before the start of the episode and ends 72 hours after its end. I cannot see any particular problems that would coincide with the episode, from that set of probes (RIPE TTM). Because we rely on default routes to our three transit providers, and Level(3) is one of them, some of our customers must have had connectivity issues to Cogent for a few hours, until we noticed (thanks to Adam Rothschild and NANOG) and implemented a workaround. But our RIPE TTM boxes (tt85 as well as the currently broken tt86) aren't in those parts of our network. I wonder if they made separate arrangements for that or are planning to make arrangements for phase 2. As someone else said, partial unreachability of a particular root nameserver isn't that much of an issue. But it's an interesting question nevertheless. -- Simon.
Re: Too much on Cogent and Level 3
I don't think anyone is learning anything new at this point. On the contrary, I think that the number of people posting with misconceptions demonstrates that many people *ARE* learning something new at this point. Especially since people who do have clue have stepped forth to try and clear up these misconceptions. This is really the only way to pass on knowledge to the next generation given that Internet routing/peering is an arcane subject which is not taught in universities. Hopefully people realize that the mailing list only contains the freshman class and they need to attend a few NANOG conferences to progress further. --Michael Dillon
RE: Cogent/Level 3 depeering (philosophical solution)
[EMAIL PROTECTED] (David Schwartz) writes: I think the industry simply needs to accept that it's more expensive to receive traffic than to send it. It is? For everybody? For always? That's a BIG statement. Can you justify? In those cases where it in fact is and there's nothing you can do about it, you need to accept it. You should not expect to be able to shift the burden of carrying your customers' traffic on your network to others. (The fact that you can sometimes bully or blackmail and get away with it doesn't justify it.) ... The question is whether the benefit to each side exceeds their cost. Yea, verily. But I don't think you'll find a one-cost-fits-all model. When one person's costs are lower than another and they're doing similar things, it's often called efficiency or competitiveness. (Just as one example.) I heartily agree. My point is simply that the your customers are getting more out of our network that our customers are argument is bull. Your customers are paying you to carry their traffic over your network. There can certainly be legitimate peering disputes about where to peer and whether there are enough peering points. If someone wants you to peer with them at just one place, it would certainly be more cost-effective for you to reach them through a transit provider you meet in multiple places, for example. (You could definitely refuse settlement-free peering if it actually increases your costs to reach the peer.) I am not making the pie-in-the-sky argument that everyone should peer with everyone else. I am specifically rejecting the argument that a traffic direction imbalance is grounds for rejecting settlement-free peering. If your customers want to receive traffic and receiving is more expensive, then that's what they're paying you for. Again, carrying *your* customers' traffic over *your* network is what *your* customers are paying *you* for. If your customers want more expensive traffic, you should bear that greater burden. A traffic direction imbalance is not reasonable grounds for rejecting SFI. The direction your customers want their traffic to go is more valuable and it's okay if it costs more. DS
Re: Cogent/Level 3 depeering (philosophical solution)
Peering Ratios? It is very timely that the upcoming NANOG Peering BOF X in Los Angeles will have a debate on this very subject: Traffic Ratios - a valid settlement metric or dinosaur from the dot.bomb past. I'm sure the strongest arguments from these threads will be clearly articulated (in a bullet point/summarized form I hope) during the debate by the debaters. At the end of the day, as with most things peering, the focus of this discussion is a meld of business and technical interests. The heat we have witnessed is probably more related to the friction of the business interests. We get very upset about the notion of fair don't we. Perhaps in the few structured minutes of the Peering BOF debate we can objectively hear both sides of this argument and provide a little light as well. Defending Traffic Ratios as a valid peering prereq: Peter Cohen Attacking Traffic Ratios as peering prereq: Richard Steenbergen Should be good fun. Bill On 10/10/05, David Schwartz [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (David Schwartz) writes: I think the industry simply needs to accept that it's more expensive to receive traffic than to send it. It is? For everybody? For always? That's a BIG statement. Can you justify? In those cases where it in fact is and there's nothing you can do about it, you need to accept it. You should not expect to be able to shift the burden of carrying your customers' traffic on your network to others. (The fact that you can sometimes bully or blackmail and get away with it doesn't justify it.) ... The question is whether the benefit to each side exceeds their cost. Yea, verily. But I don't think you'll find a one-cost-fits-all model. When one person's costs are lower than another and they're doing similar things, it's often called efficiency or competitiveness. (Just as one example.) I heartily agree. My point is simply that the your customers are getting more out of our network that our customers are argument is bull. Your customers are paying you to carry their traffic over your network. There can certainly be legitimate peering disputes about where to peer and whether there are enough peering points. If someone wants you to peer with them at just one place, it would certainly be more cost-effective for you to reach them through a transit provider you meet in multiple places, for example. (You could definitely refuse settlement-free peering if it actually increases your costs to reach the peer.) I am not making the pie-in-the-sky argument that everyone should peer with everyone else. I am specifically rejecting the argument that a traffic direction imbalance is grounds for rejecting settlement-free peering. If your customers want to receive traffic and receiving is more expensive, then that's what they're paying you for. Again, carrying *your* customers' traffic over *your* network is what *your* customers are paying *you* for. If your customers want more expensive traffic, you should bear that greater burden. A traffic direction imbalance is not reasonable grounds for rejecting SFI. The direction your customers want their traffic to go is more valuable and it's okay if it costs more. DS -- // // William B. Norton [EMAIL PROTECTED] // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Re: Cogent/Level 3 depeering (philosophical solution)
[EMAIL PROTECTED] (David Schwartz) writes: My point is simply that the your customers are getting more out of our network that our customers are argument is bull. Your customers are paying you to carry their traffic over your network. whenever you think you have a reasonable design, you can concept-test it for the internet by asking, what if six million people did this? i suspect that absent peering requirements, there would be a lot of WAN ISO-L2 and on-net ISO-L3 sold, a lot more ASN's on the hoof, and a bit less stability in the BGP core. since most of the transit ISO-L3 providers are also in the on-net ISO-L3 or WAN ISO-L2 (or both) business, the end result would be the same people getting paid the same amounts by the same other people, but called something else than what we call it now. maybe this would be better than my network is bigger!, no it ain't!, etc? -- Paul Vixie
Re: Cogent move without renumbering (was: Cogent/Level 3 depeering)
in a pay-me-now-or-pay-me-later scenario, you have to pick now vs. later. (it's a pity that the internet, for all its power, cannot alter that rule.) It should be noted that if one opts for 'later', you can do quick and dirty games with NAT. Do not renumber, change providers and put a NAT between yourself and your provider. This will continue to work until such time as your original PA space is reassigned and then you will not be able to reach the new assignee. This allows for quick moves, but creates the mortgage of an eventual renumbering. Folks who take this approach are likely to renumber into RFC 1918 space. Before you break out the blowtorches, I'm *not* claiming that this a good way of doing things. It's a hack. It's expedient. ;-) Regards, Tony
Re: Cogent/Level 3 depeering
On Sat, 8 Oct 2005, [EMAIL PROTECTED] wrote: On Sat, 08 Oct 2005 20:41:55 BST, Stephen J. Wilcox said: my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Sure, but consider is it worse to have a very small number of complaining customers who cant get to a bit of the web for 2 or 3 days, or a complete outage to the Internet for a few hours because of a problem you cant fix. I see the latter occurring quite frequently, in particular I see support queries about loss of connectivity to large parts of the Internet which on inspection was caused by dampening because the ISP was flapping. I'm just saying, you fix one problem and create a whole bunch of new ones and it depends on the customer as to which results in the optimum situation. Steve
How to multihome endusers [was: Cogent/Level 3 depeering]
Yes, indeed, I think it makes sense to multihome my humble enduser pc. Right now all I can get is aDSL and it does not matter what provider because they all use DTAG.DE infrastructure. Maybe cable will be choce. It is not as fast as aDSL at least not here and it will take another two or three years until they deploy it. If it does not get shot on site again by the regulation office or the cartell office again. So I will end up having a cable-modem speaking ethernet/PPPoE and an aDSL-modem speaking ethernet/ tcp/ip and DHCP. My ip adresses probably will be 84.167.xxx.xxx for aDSL and 24.xxx.xxx.xxx for the cable. I can talk to no-ip.com, they will allow a second ip for host_look(84.167.252.166,echnaton.serveftp.com,1420295334). host_name(84.167.252.166,p54A7FCA6.dip.t-dialin.net). Its entry will look a bit like this one: host_look(81.88.34.51,Kunden2.KONTENT.de,1364730419). host_name(81.88.34.51,kunden2-1.kontent.de). host_look(81.88.34.52,Kunden2.KONTENT.de,1364730420). host_name(81.88.34.52,kunden2-2.kontent.de). So I will end up with 3 names and 2 ip addresses for my humble host. Do I need BGP now or OSPF or can I rely on RIP. Do I need an AS number? How do I get it? Imagine not a fool like me is asking this but some 32K end users of DTAG.DE connected to a DSLAM at Franfurt/Main in germany. I guess the number of end users disconnected be Cogent and Level 3 is not much smaller. Asbestos parapluis opened. Shoot now! Peter and Karin Dambier :) [EMAIL PROTECTED] wrote: On Sat, 08 Oct 2005 20:41:55 BST, Stephen J. Wilcox said: my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Anybody want to start a pool on how many new AS numbers will get issued as a result of this tiff, and what percent will commit a BGP whoopsie that impacts more than just themselves within the first 6 months? On the other hand, I see a business opportunity to sell new customers insurance against self-inflicted gunshot wounds to the feet here. Some providers might even consider selling a managed service at a slight loss, just for self-defense.. :) -- Peter and Karin Dambier Public-Root Graeffstrasse 14 D-64646 Heppenheim +49-6252-671788 (Telekom) +49-179-108-3978 (O2 Genion) +49-6252-750308 (VoIP: sipgate.de) +1-360-448-1275 (VoIP: freeworldialup.com) mail: [EMAIL PROTECTED] http://iason.site.voila.fr http://www.kokoom.com/iason
Re: Level 3's side of the story
At 9:37 PM -0400 10/8/05, Christian Kuhtz wrote: On Oct 8, 2005, at 6:43 PM, John Curran wrote: What I have said that there is *significant* attention to the potential consumer impact of our non-essential IP services, and that's not surprising given the historic public policy in this area. I pointed to the bill under draft merely as documentation of this attention and to note that unless there is a radical shift in policy for telecom consumer protections, we are going to see some form of regulation as more voice moves to the Internet. Perhaps. Your presumption for such prediction is that service will not evolve in a disruptive fashion. No, my prediction is based solely on the current actions of lawmakers to address the perceived social need of reliable phone service. If laws are passed, it's highly likely that the FCC will regulate accordingly, regardless of how amusing the mapping from regulation to reality turns out. /John
