Re: Level 3's side of the story

[Daniel Karrenberg]

On 16.10 16:04, Simon Leinen wrote:
> 
> Kevin Loch writes:
> > Does anyone have reachability data for c-root during this episode?
> 
> The RIPE NCC "DNSMON" service has some:
> 
> http://dnsmon.ripe.net/dns-servmon/server/plot?server=c.root-servers.net&type=drops&tstart=1128246543&tstop=1128972253

If there is anyone with more comprehensive  data I'd like to hear about it ;-) !

> According to BGPlay for that particular prefix from Route-Views data
> (for some reason the RIPE RIS server used by BGPlay seems to be down
> at the moment), the "episode" seems to be between these times (UTC):

It is up now. Of the two routes to c.root-servers.net via Level 3 in that 
data set one stayed up during the period and the other got healed immediateely 
via AS286. It flipped back later.

> ...

> The interval in the URL above starts 72 hours before the start of the
> episode and ends 72 hours after its end.  I cannot see any particular
> problems that would coincide with the episode, from that set of probes
> (RIPE TTM).

I agree that there is nothing really significant here. It is mostly noise.
What one is looking for in these graphs is strong vertical 
patterns.  dnsmon did not detect anything significant concerning
reachability of c.root-servers.net.

> As someone else said, partial unreachability of a particular root
> nameserver isn't that much of an issue.  

Indeed.

> But it's an interesting question nevertheless.

A red instance of a fish from the northern seas ?

Daniel

Re: Level 3's side of the story

[Simon Leinen]

Kevin Loch writes:
> Does anyone have reachability data for c-root during this episode?

The RIPE NCC "DNSMON" service has some:

http://dnsmon.ripe.net/dns-servmon/server/plot?server=c.root-servers.net&type=drops&tstart=1128246543&tstop=1128972253

According to BGPlay for that particular prefix from Route-Views data
(for some reason the RIPE RIS server used by BGPlay seems to be down
at the moment), the "episode" seems to be between these times (UTC):

 2005-10-05 09:49:03   Route Withdrawal ( 3356 174 2149 )
 2005-10-07 19:24:13   Route Announcement   3356 174 2149

The interval in the URL above starts 72 hours before the start of the
episode and ends 72 hours after its end.  I cannot see any particular
problems that would coincide with the episode, from that set of probes
(RIPE TTM).

Because we rely on default routes to our three transit providers, and
Level(3) is one of them, some of our customers must have had
connectivity issues to Cogent for a few hours, until we noticed
(thanks to Adam Rothschild and NANOG) and implemented a workaround.
But our RIPE TTM boxes (tt85 as well as the currently broken tt86)
aren't in those parts of our network.

> I wonder if they made separate arrangements for that or are planning
> to make arrangements for phase 2.

As someone else said, partial unreachability of a particular root
nameserver isn't that much of an issue.  But it's an interesting
question nevertheless.
-- 
Simon.

Re: Level 3's side of the story

[jmalcolm]

Richard A Steenbergen writes:
>For anyone keeping score, the 
>last two times Cogent was depeered, it responded by intentionally blocking 
>connectivity to the network in question, despite the fact that both of 
>those networks were Sprint customers and thus perfectly reachable under 
>the Sprint transit Cogent gets from Verio. While no one has come forward 
>to say if the Cogent/Verio agreement is structured for full transit or 
>only Sprint/ATDN routes, Cogent has certainly set a precedent for 
>intentionally disrupting connectivity in response to depeering, as a scare 
>tactic to keep other networks from depeering them.

Without getting into the question of what is "right", it's perfectly
plausible that the Cogent-Verio interconnection is structured such
that Verio didn't send Cogent routes to FT via Sprint. Consider the
hypothetical case where Verio might peer with FT.

Joe

Re: Level 3's side of the story

[John Curran]

At 9:37 PM -0400 10/8/05, Christian Kuhtz wrote:
>On Oct 8, 2005, at 6:43 PM, John Curran wrote:
>>What I have said that there is *significant* attention to the potential 
>>consumer impact of our "non-essential" IP services, and that's not surprising 
>>given the historic public policy in this area.   I pointed to the bill under 
>>draft merely as documentation of this attention and to note that unless there 
>>is a radical shift in policy for telecom consumer protections, we are going 
>>to see some form of regulation as more voice moves to the Internet.
>
>Perhaps.  Your presumption for such prediction is that service will not evolve 
>in a disruptive fashion.

No, my prediction is based solely on the current actions of lawmakers to 
address the perceived social need of reliable phone service.  If laws are 
passed, it's highly likely that the FCC will regulate accordingly, regardless 
of how amusing the mapping from regulation to reality turns out.

/John

Re: Level 3's side of the story

[Michael Loftis]

--On October 7, 2005 7:13:45 PM -0400 William Allen Simpson 
<[EMAIL PROTECTED]> wrote:




Richard A Steenbergen wrote:

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/1
0-07-2005/0004164041&EDATE=

"On October 6, Level 3, as it had repeatedly advised Cogent it would,
terminated free traffic exchange with Cogent.  Because Internet users,
apparently without notice from Cogent and through no fault of their own,


I don't remember seeing this public notice from Level(3) posted
Wouldn't that be "without notice from Level(3)"?


Why is it (3)'s responsibility to handle Cogent's customers?  It isn't.  If 
that was the case we'd be required to notify all downstream's when we 
terminate on default of contract or other reasons (as an ISP).


No I think if (3) told Cogent then they did their job.  It's absurd to say 
that (3) was responsible for public notice of Cogent's customers or 
anything of that nature.  That's a Cogent internal matter that they screwed 
up, or intentionally withheld.

Re: Level 3's side of the story

[Patrick W. Gilmore]

On Oct 8, 2005, at 5:05 PM, Richard A Steenbergen wrote:

Because FT and Teleglobe are both full transit customers of Sprint,  
with
full global routes in, and full propagation out (this is verifiable  
via
many looking glasses). You aren't seriously going to claim that  
Cogent has

a contract with Verio which says "We will give you partial transit aka
only Sprint routes, but not Sprint routes to certain Sprint  
customers like

FT and Teleglobe", and that Cogent is throwing up its hands and saying
"sorry our contract doesn't give us routes to you, Verio won't let us
change it, what are we going to do?" are you?


Yes, I am seriously suggesting that Verio could sell 1239 +  
downstreams minus some "large" downstreams.  If I am Cogent and I  
want to get transit as cheaply as possible, I would say "don't give  
me $FOO, $BAR, $ETC, and lower your price."


Or are you seriously suggesting Cogent wouldn't do everything in its  
power to lower its costs?


Of course, "won't let us change it" is probably a bit over the top.   
I'm sure Verio will sell Cogent whatever they want.



But while we're on the subject, how do you think Sprint feels about  
Verio

selling Cogent "Sprint routes only"? I of course am not privy to exact
wording of the peering contract between Verio and Sprint (and if I  
was I
sure as hell wouldn't be talking about it on NANOG), but on many  
peering
agreements there is usually a clause that contains words like  
"shall not

give or sell nexthop to others". At the very least, it is down-right
unfriendly. Verio peers with FT and Teleglobe directly already, which
means that in order for them to send Cogent those routes via Sprint  
(which
Cogent now clearly still uses, 174 2914 1239 5511), they must have  
Cogent
directly connected on the same routers as their Sprint  
interconnections,

and have a virtual RIB set up, into which they import only the Sprint
routes. That does raise some interesting questions about how the  
contract
is written. But I agree, we're just speculating, and I'm certain no  
one is

going to give us an answer publicly.


VERY interesting.  I was completely unaware that 5511 peered directly  
with 2419.


Assuming they do, WTF would Verio not simply give Cogent direct  
routes?  Well, maybe the contract only allows Verio to propagate the  
routes to Sprint?


Or , Cogent wants to ensure FT pays for the traffic just  
like they have to pay for the traffic  


--
TTFN,
patrick

Re: Level 3's side of the story

[John Curran]

At 5:23 PM -0400 10/8/05, Christian Kuhtz wrote:
>On Oct 8, 2005, at 12:25 AM, John Curran wrote:
>>
>>That's a fine set of beliefs (and I might even agree with some of them).  
>>However, they're completely irrelevant to the existing school of thought 
>>which is guiding policy and legislation in this area, which is probably best 
>>represented by last month's House Telecom committee draft:
>>
>>   [1]
>>
>>There you go: mandatory ISP registration, interconnection, consumer 
>>protection, and more.  Maybe these folks were too busy with other stuff to 
>>notice the Internet partition happen earlier in the week?  
>>Oh, wait, I remember now: these folks are only matter when legislating.
>
>You can't be serious.  You're quoting a draft for a house bill as proof that 
>this is rational thought or justified?

Wow...   You're characterizing me as trying to prove this is rational and 
justified thought?   That's almost as cool as your earlier suggestion that I'm 
advocating for regulation.  I'm not.  I know that can be hard to tell unless 
you read my messages.

What I have said that there is *significant* attention to the potential 
consumer impact of our "non-essential" IP services, and that's not surprising 
given the historic public policy in this area.   I pointed to the bill under 
draft merely as documentation of this attention and to note that unless there 
is a radical shift in policy for telecom consumer protections, we are going to 
see some form of regulation as more voice moves to the Internet.

/John

 

Re: Level 3's side of the story

[Will Yardley]

On Sat, Oct 08, 2005 at 12:41:21PM -0400, Leo Bicknell wrote:

> Given that at least part of JC Dill's comments were directly lifted from
> an e-mail I sent him, I feel compelled to put them side by side.  JC's
> comments:

IIRC, s/him/her/

w

Re: Level 3's side of the story

[Richard A Steenbergen]

On Sat, Oct 08, 2005 at 04:37:29PM -0400, Patrick W. Gilmore wrote:
> 
> You mention at least twice (here and in the FT depeering paragraph)  
> that Cogent "accepts" the routes.
> 
> It is entirely possible, and in fact likely IMHO, that the prefixes  
> were never offered by Verio to Cogent.  Cogent pays Verio for partial  
> transit, why would Verio give Cogent more ASes than they paid for?   
> If Verio doesn't announce the prefixes, how can Cogent filter them?
> 
> Of course, I do not have enable on Cogent or Verio border routers, so  
> I cannot say for certain.  Would anyone who _knows_ care to enlighten  
> us?

Because FT and Teleglobe are both full transit customers of Sprint, with 
full global routes in, and full propagation out (this is verifiable via 
many looking glasses). You aren't seriously going to claim that Cogent has 
a contract with Verio which says "We will give you partial transit aka 
only Sprint routes, but not Sprint routes to certain Sprint customers like 
FT and Teleglobe", and that Cogent is throwing up its hands and saying 
"sorry our contract doesn't give us routes to you, Verio won't let us 
change it, what are we going to do?" are you?

But while we're on the subject, how do you think Sprint feels about Verio 
selling Cogent "Sprint routes only"? I of course am not privy to exact 
wording of the peering contract between Verio and Sprint (and if I was I 
sure as hell wouldn't be talking about it on NANOG), but on many peering 
agreements there is usually a clause that contains words like "shall not 
give or sell nexthop to others". At the very least, it is down-right 
unfriendly. Verio peers with FT and Teleglobe directly already, which 
means that in order for them to send Cogent those routes via Sprint (which 
Cogent now clearly still uses, 174 2914 1239 5511), they must have Cogent 
directly connected on the same routers as their Sprint interconnections, 
and have a virtual RIB set up, into which they import only the Sprint 
routes. That does raise some interesting questions about how the contract 
is written. But I agree, we're just speculating, and I'm certain no one is 
going to give us an answer publicly.

-- 
Richard A Steenbergen <[EMAIL PROTECTED]>   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Level 3's side of the story

[Stephen J. Wilcox]

On Sat, 8 Oct 2005, Richard A Steenbergen wrote:

> the last two times Cogent was depeered, it responded by intentionally blocking
> connectivity to the network in question, despite the fact that both of those
> networks were Sprint customers and thus perfectly reachable under the Sprint
> transit Cogent gets from Verio. While no one has come forward to say if the
> Cogent/Verio agreement is structured for full transit or only Sprint/ATDN
> routes, Cogent has certainly set a precedent for intentionally disrupting
> connectivity in response to depeering, as a scare tactic to keep other
> networks from depeering them.

i dont see it like that.. and you reapply your view in your later email to me.

cogent and level3 were peers. level3 want to change that, the only solution 
level3 will consider is for cogent to purchase transit with another provider 
(sprint/verio) or pay them direct.


whether cogent's contract with verio could provide it transit to level3 for the 
same price is irrelevant. the fact is cogent currently does not use verio for 
this and they do not want to add a number of Gbps to their transit service

theres nothing special about level3 being tier-1 and cogent being tier-2 with 
verio transit. the status of these networks is not of issue, both sides have a 
right to decide whether to connect via settlement free peering or not. of 
course 
for level3 to transit to cogent would be inconceivable to them, but thats ego / 
economics / marketing, not a principle of networking

that either network could use transit to reach the other is an engineering 
point, that neither wants to pay to do so is a business point. and this is a 
business problem.

Steve

Re: Level 3's side of the story

[Richard A Steenbergen]

On Sat, Oct 08, 2005 at 09:16:25PM +0100, Stephen J. Wilcox wrote:
> 
> understand the inability to reach cogent was the desired result for level3, 
> had 
> a contingency been put in place level3 would have been heading in the 
> opposite 
> direction to which they are moving (they are moving to force cogent to buy 
> transit, not moving to pay for their connectivity to cogent nor to keep the 
> current settlement free arrangement)

Not true. The inability to reach Cogent *DIRECTLY* was the desired result 
for (3). Everyone who has depeered Cogent so far seems genuinely astounded 
that they are perfectly willing to gamble their customers to win a peering 
dispute each and every time. I guarantee you that (3)'s desired outcome 
was that Cogent do what every other ISP who buys transit does when they 
get depeered, send the bits down the transit instead.

> cogent has not got a transit provider giving them level3 routes (as far as we 
> understand) and they have not gone and setup any such transit arrangement 
> whilst 
> waiting for the depeering.

Just to clarify the point (though I know you know this, others don't), 
Cogent does not have a transit provider CURRENTLY providing them (3) 
routes. Whether this is the result of not having a contract in place, 
asking Verio not to send them (3) routes, or simply rejecting the routes 
themselves and tagging their announcements to Verio with a no-export to 
3356 community is unknown (at least to the general public :P).

Given Cogent's position of intentional network segmentation in the two 
most recent peering disputes prior to this, in which both networks WERE 
reachable through the Sprint routes which Cogent *DID* have existing 
arragements to but which they chose not to use, it is not reasonable to 
think that the same would hold true here whether they had an existing 
ability to flip a switch and route via (3) or not.

-- 
Richard A Steenbergen <[EMAIL PROTECTED]>   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Level 3's side of the story

[Patrick W. Gilmore]

On Oct 8, 2005, at 3:07 PM, Richard A Steenbergen wrote:

Teleglobe depeers Cogent due to various ratio and market pressure  
issues.

Of note is that Cogent has recently entered the Canadian market where
Teleglobe has a strong presence, and has started giving away free or
nearly free transit to large inbound networks. Teleglobe is a Sprint
customer, and Cogent reaches Sprint through Verio. Teleglobe is caught
completely off-guard when Cogent refuses to accept the route via  
Sprint

transit, and blocks traffic between the networks. This continues for
several days, until eventually routes are leaked/added from  
Teleglobe to
SAVVIS (AS3561), who Cogent peers with. This continues for a few  
days more

until Teleglobe finally agrees to repeer Cogent.


[SNIP]

You mention at least twice (here and in the FT depeering paragraph)  
that Cogent "accepts" the routes.


It is entirely possible, and in fact likely IMHO, that the prefixes  
were never offered by Verio to Cogent.  Cogent pays Verio for partial  
transit, why would Verio give Cogent more ASes than they paid for?   
If Verio doesn't announce the prefixes, how can Cogent filter them?


Of course, I do not have enable on Cogent or Verio border routers, so  
I cannot say for certain.  Would anyone who _knows_ care to enlighten  
us?


--
TTFN,
patrick

Re: Level 3's side of the story

[Stephen J. Wilcox]

On Sat, 8 Oct 2005, Eric Louie wrote:

> DISCLAIMER:  From one of the clueless
i'm not picking on you as theres a huge amount of misinformation being posted 
on 
this thread but perhaps a private email to someone with questions might be 
better. having said that, perhaps this will serve to educate..

> During this entire debaucle, I never saw any mention of:
> 
> 1)  Cogent sending "transit" traffic to Level3, which leads me to believe that
> all the traffic from Cogent through the peering points was actually *destined*
> for Level3 customers.  Does the routing support this idea?  Is it safe to
> assume the opposite, also... that only traffic destined for Cogent customers
> came through the Level3 peering points?  And that Level3 had one and only one
> path to Cogent (no one else providing transit for them to Cogent AS'es?)

peering is all about exchanging bgp on your customers with the peer, it excludes
sending routes from another peer which is usually called transit

> 2)  Level3 making any contingency for their own customers to reach Cogent
> networks (any announcements to their own customers)

understand the inability to reach cogent was the desired result for level3, had 
a contingency been put in place level3 would have been heading in the opposite 
direction to which they are moving (they are moving to force cogent to buy 
transit, not moving to pay for their connectivity to cogent nor to keep the 
current settlement free arrangement)

> 3)  Possible traffic issues.  Was Cogent guilty of not transporting the 
> Level3-bound packets within the Cogent network to the closest point-of-entry 
> peer to the host in the Level3 network, therefore "costing" Level3 transit 
> of their own packets?  In other words is it also a traffic engineering 
> issue?

cogent has not got a transit provider giving them level3 routes (as far as we 
understand) and they have not gone and setup any such transit arrangement 
whilst 
waiting for the depeering.

it is not allowed for you to send traffic to another network via a peering. so
eg if you peer with me we will send you only our customer routes, if you
forcibly get cogents routes and route them over us without our permission you
are stealing bandwidth and we will either depeer, sue, or both. to obtain this 
'permission' would mean us acting as a transit for you and we'd probably want 
some money to do that. (consider i do not exchange money with my peers, if one 
peer uses me to reach another peer nobody is paying me for the use of my 
network).

> Are some of the business issues solvable by proper engineering and filtering
> (or statistics-jockeying)?

short answer: no, this is a political and business problem not one of 
engineering.

longer answer: level3 may be claiming that either cogent has insufficient 
traffic, that it has too much outbound or that it isnt in enough geographic 
locations. cogent could invest money to comply with level3s peering 
requirement. 
but this ultimately results in cogent spending money either to meet a new 
peering requirement or paying level3 direct to maintain a settlement peering.

Steve

> 
> - Original Message - 
> From: "Jon Lewis" <[EMAIL PROTECTED]>
> To: 
> Sent: Friday, October 07, 2005 9:45 PM
> Subject: RE: Level 3's side of the story
> 
> 
> >
> > On Fri, 7 Oct 2005, David Hubbard wrote:
> >
> >>> I don't remember seeing this public notice from Level(3) posted
> >>> Wouldn't that be "without notice from Level(3)"?
> >>
> >> They notified Cogent, not the public.  Cogent chose to
> >
> > I think it's also interesting, that AFAIK, Level3 didn't give their own 
> > customers any advance notice.  We're a customer.  I saw nothing about this 
> > until it hit nanog.  We're multi homed, so the impact on us was unnoticed.
> >
> > Suppose you're a single homed L3 or Cogent customer doing regular business 
> > with a single homed Cogent or L3 customer.  If your provider gave you 
> > several weeks notice, and if you realized the coming problem, you might 
> > take some steps to work around the issue, depending on how important your 
> > internet communications are.  Do the typical peering NDAs forbid giving 
> > customers this sort of notice?  Is it better to surprise them with a 
> > multi-day outage and then give them 30 days notice that it's going to 
> > happen again??
> >
> >>> Splendid, that gives the world sufficient time to accept
> >>> Cogent's offer of 1 year free service.
> >>
> >> This is not the first time Cogent has used their customers
> >> as pawns in peering disputes, I don't know if I'd jump on
> >&g

Re: Level 3's side of the story

[Matthew Crocker]

Level 3 claims Cogent is sending far more traffic than Level3 to  
Cogent.
Thus, Level3's viewpoint is that Cogent relies on them more than  
they rely
on Cogent.  Thus, it no longer makes sense in their view point to  
maintain
a free interconnection as there is no similar balance of traffic  
ratio.




This has  always bugged me.  Is a Cogent customer sending traffic to  
a L3 customer or is a L3 customer requesting the traffic from a  
Cogent customer?  Traffic is traffic,  L3 has eyeballs,  Cogent has  
content producers.  Of course most of the traffic will flow from  
Cogent -> L3.  L3 chose to sell to eyeball customers, Cogent chose to  
sell to content producers.  If the L3 customers didn't create the  
demand for the traffic then I'm sure Cogent wouldn't be sending them  
the traffic.


IMHO the only valid complaint L3 has is wether Cogent is hot-potato  
routing the traffic causing L3 to 'incur more cost'.  That should all  
be spelled out in the peering agreement.


--
Matthew S. Crocker
Vice President
Crocker Communications, Inc.
Internet Division
PO BOX 710
Greenfield, MA 01302-0710
http://www.crocker.com

Re: Level 3's side of the story

[Richard A Steenbergen]

On Sat, Oct 08, 2005 at 07:24:06AM -0700, JC Dill wrote:
> 
> Cogent was a "tier 1" until prior de-peering incidents left them unable 
> to reach other networks.  They solved this by buying filtered transit 
> thru Verio to reach the networks they couldn't reach via peering.

For the record, Cogent was never a Tier 1. They have never had Sprint 
peering (unless you count the 30 seconds between acquisition of a company 
that did have it, and the depeering notice, years ago). Cogent's history 
of depeering debacles, at least as best as I can remember them, is:

ATDN (AS1668) depeers Cogent, December 18 2002.
http://www.cctec.com/maillists/nanog/historical/0212/msg00366.html
http://www.cctec.com/maillists/nanog/historical/0212/msg00412.html

ATDN is in the process of shutting off legacy transit and peering on its 
path to tier-1-dom, and disconnects Cogent due to ratio (also technically 
Cogent is still on a trial peering session). At this time, Cogent is a 
full transit customer of AboveNet (AS6461), ATDN is still a full transit 
customer of Level 3, and Cogent is a peer of Level 3. Following the 
depeering, Cogent shifts 100% of the traffic to their (3) peers, which 
become severely congested nearly 24/7 for several weeks. Despite being 
able to send some traffic to AboveNet transit, they decide to leave 
traffic congested to (3) to see if ATDN will repeer (not knowing that AOL 
customers don't know what peering is and thus won't be nearly as vocal as 
Cogent's customers). Traffic stays congested until Cogent's peering 
capacity with (3) is upgraded. ATDN later switches their routing with (3) 
from transit to customer-only routes (removing the last of their transit 
paths), at which point Cogent shifts traffic to newly acquired Verio 
transit to reach them.



Teleglobe (AS6453) depeers Cogent, some time in Feb 2005?
Don't ask me why but I can't find a NANOG thread discussing this.

Teleglobe depeers Cogent due to various ratio and market pressure issues. 
Of note is that Cogent has recently entered the Canadian market where 
Teleglobe has a strong presence, and has started giving away free or 
nearly free transit to large inbound networks. Teleglobe is a Sprint 
customer, and Cogent reaches Sprint through Verio. Teleglobe is caught 
completely off-guard when Cogent refuses to accept the route via Sprint 
transit, and blocks traffic between the networks. This continues for 
several days, until eventually routes are leaked/added from Teleglobe to 
SAVVIS (AS3561), who Cogent peers with. This continues for a few days more 
until Teleglobe finally agrees to repeer Cogent.



France Telecom (OpenTransit/AS5511) depeers Cogent, April 14 2005
http://www.merit.edu/mail.archives/nanog/2005-04/msg00484.html

FT depeers Cogent due to, well, a variety of issues and general 
unhappiness surrounding Cogent's entrance into their markets through the 
purchase of Lambdanet. FT is a Sprint customer, Cogent is already 
receiving Sprint routes via Verio but intentionally blocks these routes so 
that they have no path to FT. The rumored resolution to the dispute is 
that a FT customer sues Cogent in France, and a French judge either does 
or is about to fine the hell out of Cogent unless connectivity is 
restored. At this point Cogent caves, and begins accepting the routes via 
Sprint (via Verio).



Of course I am certain there are a lot more depeerings (both from and to 
Cogent) that did not make the news, but these are the big notable events 
that dramatically impacted connectivity. For anyone keeping score, the 
last two times Cogent was depeered, it responded by intentionally blocking 
connectivity to the network in question, despite the fact that both of 
those networks were Sprint customers and thus perfectly reachable under 
the Sprint transit Cogent gets from Verio. While no one has come forward 
to say if the Cogent/Verio agreement is structured for full transit or 
only Sprint/ATDN routes, Cogent has certainly set a precedent for 
intentionally disrupting connectivity in response to depeering, as a scare 
tactic to keep other networks from depeering them.


> L3 was hoping to force Cogent to increase that transit to include the 
> traffic destined for L3's customers, thus raising Cogent's transport 
> costs at no additional (transport) cost to L3.

As I've already pointed out, L3 depeering Cogent is in fact a major 
revenue loss for L3. Not only will they not make any money off of Cogent 
(since we both know Cogent will NEVER give them money for direct transit), 
but Cogent will heavily depref them and shift many many gigabits of 
traffic away from L3 and onto their competitors, traffic that L3 was 
previously billing their customers for. They'll also lose customers during 
the unreachability, and even if Cogent buckles and buys transit they'll 
lose some outbound traffic from their multihomed customers due to a longer 
as-path length to reach Cogent and many of Cogent's routes (11k of them 
remember).

Let me be p

Re: Level 3's side of the story

[Leo Bicknell]

Given that at least part of JC Dill's comments were directly lifted from
an e-mail I sent him, I feel compelled to put them side by side.  JC's
comments:

In a message written on Sat, Oct 08, 2005 at 07:24:06AM -0700, JC Dill wrote:
> Consider a simple hypothetical closest-exited network setup (hot potato 
> routing) between 2 peers:
> 
> 
> ISP Eyeballs:  Router-E12,000 Mile LinkRouter-E2Customer
>  | |
>  | |
>peering   peering
>   | |
>   | |
> ISP Content: Server--Router-C1---2,000 Mile Link-Router-C2
> 
> When the customer on ISP E (Eyeballs) requests content (web page, music 
> file, etc.) from the server on ISP C (Content), packets travel like this:
> 
> Customer->Router-E2->Router-C2->Router-C1->Server
> 
> When the server returns traffic to the customer, traffic goes like this:
> 
> Server->Router-C1->Router-E1->Router-E2->Customer
> 
> The problem is the customer->server direction would typically be a 500 
> byte request and 64 byte ACK packets, where as the server->customer data 
> includes many 1500 byte data packets.  So, ISP Eyeballs may carry 2Mbs 
> of data over its 2,000 mile link, where as ISP Content will only carry 
> 128Kbs over its 2,000 mile link.
> 
> Even though both companies met in the middle, ISP Content shifted  some 
> of its costs to ISP Eyeballs.
> 
> Back when most ISPs had the same types of traffic (even mixes of content 
> and eyeballs), they had even ratios which equalized this effect, as it 
> was happening the same amount in both directions.  But as some ISPs 
> started specializing in one type of content or the other, uneven flows 
> were produced.  Some bean-counter felt that these uneven flows meant 
> that the network that was sending more traffic should now pay for 
> transit, even though this traffic was traffic that their own customers 
> were requesting and paying them to transmit!
> 
> There are ways to deal with it though, like cold potato routing.

My message to JC:

In a message written on Thu, 6 Oct 2005 15:07:05 -0400, Leo Bicknell wrote:
> That's not how it works.  Consider a simple, closest exited network
> setup:
> 
> 
> ISP A:   Router1---2,000 Mile Link-Router2Customer
> | |
> | |
>  peering   peering
> | |
> | |
> ISP B: ServerRouterA---2,000 Mile Link-RouterB
> 
> When the "customer" requests a web page from the "server", packets
> travel like this:
> 
> Customer->Router2->RouterB->RouterA->Server
> 
> When the server returns traffic to the customer, traffic goes like this:
> 
> Server->RouterA->Router1->Router2->Customer
> 
> The problem is the customer->server direction is a 500 byte request,
> and 64 byte ACK packets, where as the server->customer data includes
> lots of 1500 byte data packets.  So, ISP A may carry 2Mbps of data
> over it's 2,000 mile link, where as ISP B will only carry 128k over
> it's 2.000 mile link.
> 
> Even though both companies met in the middle, ISP B shifted it's costs
> to ISP A.
> 
> The theory is that having an even ratio equalizes this effect, as it's
> happening the same amount in both directions.  There are other ways to
> deal with it though, like cold potato routing and other tricks.
> 
> In practice it becomes a much more complicated issue, but in many cases
> due to how routing works, geographies involved, and others routing
> policies (eg, customers not advertising their routes to all providers)
> there are very real, very expensive inequalities.  Large ISP's work
> together to equalize them to the extent possible.

It's not so much that I mind quoting without attribution, it's that
I mind interleaving original bits with new bits as that can lead
to confusion later least anyone put A and B together.

On the issue at hand, surface it to say costs are a complicated issue.
This is a rather simplified view of the world, and does not take into
consideration many of the things that are going on.

For instance, many content providers buy cheap Cogent bandwidth and
dump traffic on Cogent, but DO NOT advertise ANY prefixes to Cogent,
or at least, not the ones generating traffic.  So in my example if
we assume "customer" is Level 3, and "Server" is someone buying
from Cogent, the path from customer to server may not transit
Cogent's network at all.  Level 3 may send that to say, AT&T, who's
also a service provider for the person with the server.

There's also a lot of other considerations that come down to various
people's choices.  How many people bought ATM cards and had them
b

RE: Level 3's side of the story

[Rik van Riel]

On Sat, 8 Oct 2005, Jon Lewis wrote:

> If you're multihomed and using Cogent as a cheap bandwidth whore, does 
> it matter if their cheap bandwidth gives you 155k routes instead of 168k 
> routes? After all, if its cheap and off-loads enough traffic from your 
> more expensive 168k route circuits, isn't it doing what you bought it 
> for?

I wonder if Cogent customers can get their money back if more
than 0.1% of their packets go to L3 single-homed networks ;)


http://www.cogentco.com/htdocs/internet.php?current=10

Product Features:

FULL SLA: Our SLA guarantees 99.99% network availability, 99.9% packet 
delivery, less than 50 milliseconds roundtrip latency and proactive outage 
notification within 15 minutes.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: Level 3's side of the story

[Patrick W. Gilmore]

On Oct 8, 2005, at 11:42 AM, John Curran wrote:

Cold potato routing alone is insufficient in many cases, and some  
form of

settlement becomes necessary.

http://www.cctec.com/maillists/nanog/historical/9808/msg00517.html


This does not in any way explain why cold-potato routing is  
insufficient.


But I seriously doubt a consensus will be found in this forum, or  
between John & me. :)


Let's just say that there _are_ technical means to make "traffic  
imbalances" either (close to) fair, or reverse the financial burden.   
And almost every hoster I know (and certainly some of the ones  
involved in the original BBN / Exodus / Above.Net / etc. fracas) were  
perfectly willing to take on more of the cost so the eyeball network  
actually has a financial incentive to peer.


--
TTFN,
patrick

Re: Level 3's side of the story

[John Curran]

At 10:24 AM -0400 10/8/05, James wrote:
>
> > 3)  Possible traffic issues.  Was Cogent guilty of not transporting the
>> Level3-bound packets within the Cogent network to the closest
>> point-of-entry peer to the host in the Level3 network, therefore "costing"
>> Level3 transit of their own packets?  In other words is it also a traffic
>> engineering issue?
>
>No, that was BBN vs. Exodus depeering superbowl back in 90's, which was
>resolved through cold potato routing.

*cough*

Cold potato routing alone is insufficient in many cases, and some form of
settlement becomes necessary.

http://www.cctec.com/maillists/nanog/historical/9808/msg00517.html

/John

Re: Level 3's side of the story

[James]

On Sat, Oct 08, 2005 at 04:54:52AM -0700, Eric Louie wrote:
> 
> DISCLAIMER:  From one of the clueless
> 
> During this entire debaucle, I never saw any mention of:
> 
> 1)  Cogent sending "transit" traffic to Level3, which leads me to believe 
> that all the traffic from Cogent through the peering points was actually 
> *destined* for Level3 customers.   Does the routing support this idea?  Is 
> it safe to assume the opposite, also... that only traffic destined for 
> Cogent customers came through the Level3 peering points?  And that Level3 
> had one and only one path to Cogent (no one else providing transit for them 
> to Cogent AS'es?)

Neither Level3 nor Cogent utilize a middleman called "transit" to reach each
other.  Sure, Cogent has Verio as its transit, but Verio is used as a partial
transit to only reach some spot nets (e.g. Sprint, France Telecom, ATDN, etc).

You may want to better understand the difference between a transit 
relationship and a peering relationship.  Here is exerpt from wbn's Equinix
paper:

8<---
Transit is the business relationship whereby one ISP provides (usually sells) 
access to all destinations in its routing table.
--->8
(Note: Also several ISPs offer "regional transit" or "limited/partial transit", 
by providing regional routes instead of full routing table access.  Cogent's 
Verio transit service is partial transit.).

8<---
Peering is the business relationship whereby ISPs reciprocally provide access 
to each others' customers (no access to whole internet is provided).
--->8

> 
> 2)  Level3 making any contingency for their own customers to reach Cogent 
> networks (any announcements to their own customers)

This is unheard of unfortunately to date.  But as mentioned, Cogent has made
numerous contacts through sales channel to accomodate customer needs before
the disconnection.

> 
> 3)  Possible traffic issues.  Was Cogent guilty of not transporting the 
> Level3-bound packets within the Cogent network to the closest 
> point-of-entry peer to the host in the Level3 network, therefore "costing" 
> Level3 transit of their own packets?  In other words is it also a traffic 
> engineering issue?

No, that was BBN vs. Exodus depeering superbowl back in 90's, which was
resolved through cold potato routing.

Level 3 claims Cogent is sending far more traffic than Level3 to Cogent.
Thus, Level3's viewpoint is that Cogent relies on them more than they rely
on Cogent.  Thus, it no longer makes sense in their view point to maintain
a free interconnection as there is no similar balance of traffic ratio.

Cogent claims Level3 depeered because of competingly low market prices for
transit bandwidth sold by Cogent.  Cogent further claims that it is Level3
who requested Cogent to send more traffic.  So... which side is really true,
no one knows at this point, but these are what both sides are stating.

> 
> Are some of the business issues solvable by proper engineering and 
> filtering (or statistics-jockeying)?

Yes and no.  Cogent could technically deprefer Level3 peering partially or
totally to reduce the egress traffic to Level3 peering.  Since many of Level3
customers are multihomed, deprefering Level3 peering to certain extent would
cause significant chunk of bits to reroute to Cogent's other peers who are
providing additional transit to said customers.

But you have to realize though that often times, significant peering disputes
arise from business and political related issues, not technical.  There are
lot of technical peering disputes going on everyday, and majority of those
can be resolved quietly without even impacting users other than a "scheduled
maintenance" event.

The current depeering dispute is really more or less "desperation depeering"
as Richard pointed out.  Thus it is not necessarily initiated by technical 
disputes that can be resolved using traffic engineering, but trying to see
who blinks and pays up.  Some may call it "extortion", some may call it
"the depeeree deserved it" or some may call it "both sides burning bridges."
and other views, etc..


James

-- 
James Jun
Infrastructure and Technology Services
TowardEX Technologies
Office +1-617-459-4051 x179 | Mobile +1-978-394-2867
[EMAIL PROTECTED] | www.towardex.com

Re: Level 3's side of the story

[JC Dill]

Eric Louie wrote:


DISCLAIMER:  From one of the clueless


As a disclaimer, I will point out that there are some in this debate who 
consider me clueless as well.  However, I don't believe that any of the 
following is in error.



During this entire debaucle, I never saw any mention of:


I've mentioned most of it in prior posts but perhaps it hasn't been 
explained well enough for you to fully grasp it.  I hope this helps.


1)  Cogent sending "transit" traffic to Level3, which leads me to 


Cogent was sending peering traffic, not transit traffic.

believe that all the traffic from Cogent through the peering points was 
actually *destined* for Level3 customers.   


Yes, that's what peering is. Peering gets you the other ISP's customers 
but NOT all the rest of the internet that is reachable thru that ISP. 
Transit gets you all the internet that is reachable thru that ISP.


Does the routing support this idea?  


Yes.

Is it safe to assume the opposite, also... that only traffic 
destined for Cogent customers came through the Level3 peering points?  


Yes.

And that Level3 had one and only one path to Cogent (no one else 
providing transit for them to Cogent AS'es?)


Yes.

2)  Level3 making any contingency for their own customers to reach 
Cogent networks (any announcements to their own customers)


L3 made no contingency plans for their customers to reach Cogent (and 
visa versa).


For the L3 customers that were multi-homed, they had other paths to 
Cogent thru their other service providers.  Those that were not 
multi-homed had no path to Cogent when L3 cut the peering, because as a 
"tier 1" (the widely accepted definition of "tier-1" is that the network 
only has SFI (Settlement Free Interchange i.e. peering) links to other 
networks and thus no transit links) none of the other networks L3 
connects to will carry L3's traffic to a third network (another peer).


Cogent was a "tier 1" until prior de-peering incidents left them unable 
to reach other networks.  They solved this by buying filtered transit 
thru Verio to reach the networks they couldn't reach via peering.


L3 was hoping to force Cogent to increase that transit to include the 
traffic destined for L3's customers, thus raising Cogent's transport 
costs at no additional (transport) cost to L3.


3)  Possible traffic issues.  Was Cogent guilty of not transporting the 
Level3-bound packets within the Cogent network to the closest 
point-of-entry peer to the host in the Level3 network, therefore 
"costing" Level3 transit of their own packets?  


Possible, in fact probable.  Most ISPs hand off traffic to peers under a 
"hot potato" policy, they hand it off at the closest point where they 
connect.  If the traffic is equal in both directions then this works. 
If the traffic is not equal, then this lowers the cost of the network 
that has high outbound traffic, as the other network bears the brunt of 
the total cost for transporting the combined traffic between their 
respective customers.



In other words is it also a traffic engineering issue?


Or rather, that it could be mostly solved with a traffic engineering fix 
called "cold potato" routing, where the sending network carries the 
traffic as far as they can before handing it off to the recipient network.


Consider a simple hypothetical closest-exited network setup (hot potato 
routing) between 2 peers:



ISP Eyeballs:  Router-E12,000 Mile LinkRouter-E2Customer
 | |
 | |
   peering   peering
  | |
  | |
ISP Content: Server--Router-C1---2,000 Mile Link-Router-C2

When the customer on ISP E (Eyeballs) requests content (web page, music 
file, etc.) from the server on ISP C (Content), packets travel like this:


Customer->Router-E2->Router-C2->Router-C1->Server

When the server returns traffic to the customer, traffic goes like this:

Server->Router-C1->Router-E1->Router-E2->Customer

The problem is the customer->server direction would typically be a 500 
byte request and 64 byte ACK packets, where as the server->customer data 
includes many 1500 byte data packets.  So, ISP Eyeballs may carry 2Mbs 
of data over its 2,000 mile link, where as ISP Content will only carry 
128Kbs over its 2,000 mile link.


Even though both companies met in the middle, ISP Content shifted  some 
of its costs to ISP Eyeballs.


Back when most ISPs had the same types of traffic (even mixes of content 
and eyeballs), they had even ratios which equalized this effect, as it 
was happening the same amount in both directions.  But as some ISPs 
started specializing in one type of content or the other, uneven flows 
were produced.  Some bean-counter felt that these uneven flows meant 
that the network that was sending more traffic should now pay for 

Re: Level 3's side of the story

[Robert E . Seastrom]

Kevin Loch <[EMAIL PROTECTED]> writes:

> Richard A Steenbergen wrote:
>> Certainly these are high-margin but low-bandwidth customers, maybe
>> with enough complaints Cogent will be willing to stick them on a
>> smaller seperate ASN which is willing to buy transit.
>
> Does anyone have reachability data for c-root during this episode?
>
> I wonder if they made separate arrangements for that or are planning
> to make arrangements for phase 2.

The nice thing about having 13 root nameservers is that it really
doesn't matter whether they do or not.

If it starts to become trendy to have root nameservers on
intentionally-partially-connected networks, (say, maybe four of them)
*then* we can start thinking about the consequences.

---rob

Re: Level 3's side of the story

[Eric Louie]

DISCLAIMER:  From one of the clueless

During this entire debaucle, I never saw any mention of:

1)  Cogent sending "transit" traffic to Level3, which leads me to believe 
that all the traffic from Cogent through the peering points was actually 
*destined* for Level3 customers.   Does the routing support this idea?  Is 
it safe to assume the opposite, also... that only traffic destined for 
Cogent customers came through the Level3 peering points?  And that Level3 
had one and only one path to Cogent (no one else providing transit for them 
to Cogent AS'es?)


2)  Level3 making any contingency for their own customers to reach Cogent 
networks (any announcements to their own customers)


3)  Possible traffic issues.  Was Cogent guilty of not transporting the 
Level3-bound packets within the Cogent network to the closest point-of-entry 
peer to the host in the Level3 network, therefore "costing" Level3 transit 
of their own packets?  In other words is it also a traffic engineering 
issue?


Are some of the business issues solvable by proper engineering and filtering 
(or statistics-jockeying)?


- Original Message - 
From: "Jon Lewis" <[EMAIL PROTECTED]>

To: 
Sent: Friday, October 07, 2005 9:45 PM
Subject: RE: Level 3's side of the story




On Fri, 7 Oct 2005, David Hubbard wrote:


I don't remember seeing this public notice from Level(3) posted
Wouldn't that be "without notice from Level(3)"?


They notified Cogent, not the public.  Cogent chose to


I think it's also interesting, that AFAIK, Level3 didn't give their own 
customers any advance notice.  We're a customer.  I saw nothing about this 
until it hit nanog.  We're multi homed, so the impact on us was unnoticed.


Suppose you're a single homed L3 or Cogent customer doing regular business 
with a single homed Cogent or L3 customer.  If your provider gave you 
several weeks notice, and if you realized the coming problem, you might 
take some steps to work around the issue, depending on how important your 
internet communications are.  Do the typical peering NDAs forbid giving 
customers this sort of notice?  Is it better to surprise them with a 
multi-day outage and then give them 30 days notice that it's going to 
happen again??



Splendid, that gives the world sufficient time to accept
Cogent's offer of 1 year free service.


This is not the first time Cogent has used their customers
as pawns in peering disputes, I don't know if I'd jump on
the bandwagon so quickly (spoken as a customer of both
companies).


If you're multihomed and using Cogent as a cheap bandwidth whore, does it 
matter if their cheap bandwidth gives you 155k routes instead of 168k 
routes?  After all, if its cheap and off-loads enough traffic from your 
more expensive 168k route circuits, isn't it doing what you bought it for?


Also, is 30 days really enough time for anyone to get a free connection to 
Cogent?  I mean if you're in a building they're already in, and its just a 
cross connect, sure that can be done quickly...but at least around here, 
getting any sort of high bandwidth circuit (>T1) can take months.  IIRC, 
the UNE DS3 connecting our office to the rest of our network was several 
months late.


--
 Jon Lewis   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net| _ http://www.lewis.org/~jlewis/pgp 
for PGP public key_



--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.11.13/124 - Release Date: 10/7/2005






--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.11.13/124 - Release Date: 10/7/2005

Re: Level 3's side of the story

[Richard A Steenbergen]

On Sat, Oct 08, 2005 at 01:24:40AM -0400, Richard A Steenbergen wrote:
> 
> I don't know how you missed it, but as far as I can tell every sales rep 
> (3) has was mobilized to call every customer or potential customer they 

Errr every sales rep Cogent had, sorry. I never heard anything about it 
from (3) or any (3) customers, for what thats worth.

-- 
Richard A Steenbergen <[EMAIL PROTECTED]>   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Level 3's side of the story

[Richard A Steenbergen]

On Sat, Oct 08, 2005 at 12:45:32AM -0400, Jon Lewis wrote:
> 
> On Fri, 7 Oct 2005, David Hubbard wrote:
> 
> >>I don't remember seeing this public notice from Level(3) posted
> >>Wouldn't that be "without notice from Level(3)"?
> >
> >They notified Cogent, not the public.  Cogent chose to
> 
> I think it's also interesting, that AFAIK, Level3 didn't give their own 
> customers any advance notice.  We're a customer.  I saw nothing about this 
> until it hit nanog.  We're multi homed, so the impact on us was unnoticed.

I don't know how you missed it, but as far as I can tell every sales rep 
(3) has was mobilized to call every customer or potential customer they 
have ever spoken to during the month of Augusst, informing them about the 
receipt of a depeering notice from Level 3 and offering affected customers 
zero-commit Cogent ports.

Among the people I know who took that offer, Cogent actually worked 
quickly (quicker than normal I would say) to get service up before the 
event. There was also a post to NANOG about the depeering in early 
September. While Cogent may not have taken out a full page ad in the New 
York Times for it, they certainly made every effort to inform those who 
needed to know, and they were never ambiguous about the fact that they 
were fully expecting to be segmented from (3).

-- 
Richard A Steenbergen <[EMAIL PROTECTED]>   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

RE: Level 3's side of the story

[Jon Lewis]

On Fri, 7 Oct 2005, David Hubbard wrote:


I don't remember seeing this public notice from Level(3) posted
Wouldn't that be "without notice from Level(3)"?


They notified Cogent, not the public.  Cogent chose to


I think it's also interesting, that AFAIK, Level3 didn't give their own 
customers any advance notice.  We're a customer.  I saw nothing about this 
until it hit nanog.  We're multi homed, so the impact on us was unnoticed.


Suppose you're a single homed L3 or Cogent customer doing regular business 
with a single homed Cogent or L3 customer.  If your provider gave you 
several weeks notice, and if you realized the coming problem, you might 
take some steps to work around the issue, depending on how important your 
internet communications are.  Do the typical peering NDAs forbid giving 
customers this sort of notice?  Is it better to surprise them with a 
multi-day outage and then give them 30 days notice that it's going to 
happen again??



Splendid, that gives the world sufficient time to accept
Cogent's offer of 1 year free service.


This is not the first time Cogent has used their customers
as pawns in peering disputes, I don't know if I'd jump on
the bandwagon so quickly (spoken as a customer of both
companies).


If you're multihomed and using Cogent as a cheap bandwidth whore, does it 
matter if their cheap bandwidth gives you 155k routes instead of 168k 
routes?  After all, if its cheap and off-loads enough traffic from your 
more expensive 168k route circuits, isn't it doing what you bought it for?


Also, is 30 days really enough time for anyone to get a free connection to 
Cogent?  I mean if you're in a building they're already in, and its just a 
cross connect, sure that can be done quickly...but at least around here, 
getting any sort of high bandwidth circuit (>T1) can take months.  IIRC, 
the UNE DS3 connecting our office to the rest of our network was several 
months late.


--
 Jon Lewis   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net| 
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: Level 3's side of the story

[John Curran]

At 11:52 PM -0400 10/7/05, Christian Kuhtz wrote:
>Sorry, I fundamentally disagree with the point of view that we need to 
>regulate free entry & free exit and throw these basic tenets of our economy 
>out the window to encourage continued poor decision making .

I'm not advocating that we need to regulate anything, only pointing out that 
the historic public policy position discourages service impacts to 3rd parties, 
and even more so when done without notice.  If you carry essential consumer 
services, then there is already existing policy in this area.

>There is no paradise behind that door.  And I don't believe regulation should 
>be used to prevent people from making occasionally poor decisions, which in 
>turn will ultimately do nothing more than raise the price floor and create 
>additional barriers of entry in the market.  If you don't have those events 
>happen, bad things never get weeded out.  There really is nothing but bad news 
>in store for what you propose, because it isn't as simple of a thing to impose 
>regulation and have mostly good things happen.

That's a fine set of beliefs (and I might even agree with some of them).  
However, they're completely irrelevant to the existing school of thought which 
is guiding policy and legislation in this area, which is probably best 
represented by last month's House Telecom committee draft: 

   [1]

There you go: mandatory ISP registration, interconnection, consumer protection, 
and more.  Maybe these folks were too busy with other stuff to notice the 
Internet partition happen earlier in the week?  Oh, wait, I remember now: these 
folks are only matter when legislating.

/John

[1] A readable (only-slightly-biased) summary:  

Re: Level 3's side of the story

[Kevin Loch]

Richard A Steenbergen wrote:
Certainly these are high-margin but low-bandwidth customers, maybe 
with enough complaints Cogent will be willing to stick them on a smaller 
seperate ASN which is willing to buy transit.


Does anyone have reachability data for c-root during this episode?

I wonder if they made separate arrangements for that or are planning to 
make arrangements for phase 2.


- Kevin

Re: Level 3's side of the story

[Richard A Steenbergen]

On Fri, Oct 07, 2005 at 11:27:09PM -0400, Patrick W. Gilmore wrote:
> 
> Is there any particular reason you think Level 3 will buy transit?   
> Or that they give a rat's ass what tier _you_ think they are?
> 
> Perhaps Cogent will have had time to buy transit?
> 
> Or perhaps - just perhaps - the Internet will be bifurcated for the  
> time being.  There is nothing intrinsically wrong with an Internet  
> which isn't fully connected.   Each business will work out its own  
> business needs and use the transit provider(s) they see fit.
> 
> I am a bit confused about why so many people think this is such a bad  
> thing.  Personally, I believe there is room in the market for partial  
> transit providers.  I guess that theory is about to be tested.

My advice to these people is, if you don't like the connectivity being 
provided, don't buy from that provider. Closing your wallet is a lot more 
powerful than flapping your lips.

Personally I suspect there will still be an extensive market of people who 
are multihomed to Cogent and someone else, and that these people will 
continue to buy the product even without the 2.5% of the Internet that is 
single homed to (3). The biggest complainers so far seem to be from the 
people who didn't understand what they were signing on for, and a lot of 
those are the Verio customers who Cogent purchased. Maybe Cogent is 
willing to part with those customers, maybe their contracts are coming due 
and they won't be renewing, or maybe the customers should have thought 
better before signing a contract which allowed the transfer of the 
contract to a completely different entity without an opportunity to 
cancel. Certainly these are high-margin but low-bandwidth customers, maybe 
with enough complaints Cogent will be willing to stick them on a smaller 
seperate ASN which is willing to buy transit.

Or, if you really care about it that much, buy service from someone else, 
unplug your port, stop paying your bill, and maybe send back a nastygram 
which includes suggestions as to where Cogent can relocate the router card 
that you used to be attached to, along with helpful advice as to how it 
might be made to fit there. Cogent will not waste time or money on 
lawsuits for disgruntled ex-Verio T1 customers.

-- 
Richard A Steenbergen <[EMAIL PROTECTED]>   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Level 3's side of the story

[Ken Leland]

On Fri, Oct 07, 2005 at 11:27:09PM -0400, Patrick W. Gilmore wrote:
> 
> I am a bit confused about why so many people think this is such a bad  
> thing.  Personally, I believe there is room in the market for partial  
> transit providers.  I guess that theory is about to be tested.
> 

Agree that there is room in the market for attactively priced
partial transit providers. Now where does that leave Level 3
after Nov 9th?  Possibly with a second strategy reconsidertion.

Ken Leland
Monmouth Telecom

Re: Level 3's side of the story

[John Curran]

At 11:00 PM -0400 10/7/05, Christian Kuhtz wrote:
>What's your point?  Are you seriously suggesting this should be regulated?

It doesn't appear necessary at all; industry self-regulation seems to be 
working just fine here... 
(one might claim almost too well :-) )

>.. let the customers draw their own consequences and conclusions based on the 
>actions of the companies who serve them.  
>Thankfully, Internet connectivity is ubiquitous for the most part, and this is 
>hardly the same as essential services (those who consider it that, are fools 
>not to have diversity).

Those un-essential Internet services increasingly carry voice services that are 
considered essential by many consumers, and for whom diversity is not a 
realistic option.  Leaving them completely open to the consequences of service 
provider actions would be quite an innovative curve in public policy.

/John

Re: Level 3's side of the story

[Patrick W. Gilmore]

On Oct 7, 2005, at 7:13 PM, William Allen Simpson wrote:

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/ 
story/10-07-2005/0004164041&EDATE=
"On October 6, Level 3, as it had repeatedly advised Cogent it  
would, terminated free traffic exchange with Cogent.  Because  
Internet users, apparently without notice from Cogent and through  
no fault of their own,


I don't remember seeing this public notice from Level(3) posted
Wouldn't that be "without notice from Level(3)"?


Every peering agreement I have seen includes an NDA.  Most certainly  
notified Cogent, and almost certainly had no legal right to issue a  
press release.



have been impacted, Level 3 has, effective immediately, re- 
established a free connection to Cogent.  In order to allow  
Internet users to make alternative arrangements,


Splendid, that gives the world sufficient time to accept Cogent's  
offer

of 1 year free service.


I note that connecting to Cogent after November 9th will still give  
you only partial transit to the Internet - in fact, far less of the  
Internet than connecting to Level 3.  (Despite Cogent's claims  
otherwise.)



we will maintain this connection until 6:00 a.m. ET, November 9,  
2005.  The effectiveness of this arrangement of course depends on  
Cogent's willingness to maintain their side of the traffic exchange.



At which time Level(3) will have had time to purchase transit, as it
will be a "tier 2" hoisted on its own petard.


Is there any particular reason you think Level 3 will buy transit?   
Or that they give a rat's ass what tier _you_ think they are?


Perhaps Cogent will have had time to buy transit?

Or perhaps - just perhaps - the Internet will be bifurcated for the  
time being.  There is nothing intrinsically wrong with an Internet  
which isn't fully connected.   Each business will work out its own  
business needs and use the transit provider(s) they see fit.


I am a bit confused about why so many people think this is such a bad  
thing.  Personally, I believe there is room in the market for partial  
transit providers.  I guess that theory is about to be tested.


--
TTFN,
patrick

Re: Level 3's side of the story

[John Curran]

At 10:15 PM -0400 10/7/05, Christian Kuhtz wrote:
>>
>>In any industry, it only takes a handful of consumers impacted through no 
>>fault of their own to generate significant political & regulatory attention.  
>>Reasonable-sounding explanations cannot stand if they imply that thousands of 
>>innocent parties can be impacted without notice.
>
>Nonsense.  That only matters when somebody is looking for an opportunity to 
>legislate.

You're right:  administrators, regulators, and representatives are completely 
helpless except when making laws, and the reconnection of services (ref: 
"Because Internet users, apparently without notice from Cogent and through no 
fault of their own, have been impacted, Level 3 has, effective immediately, 
re-established a free connection to Cogent.") can just as easily be attributed 
to uncanny judgement on Level 3's part as any other explanation.

/John

Re: Level 3's side of the story

[John Curran]

At 4:41 PM -0700 10/7/05, william(at)elan.net wrote:
>BTW - it sounds like maybe somebody was required to give 30 days notice of 
>service changes to certain customers with good laywers

In any industry, it only takes a handful of consumers impacted through no fault 
of their own to generate significant political & regulatory attention.  
Reasonable-sounding explanations cannot stand if they imply that thousands of 
innocent parties can be impacted without notice.

/John

Re: Level 3's side of the story

[Deepak Jain]

william(at)elan.net wrote:



BTW - it sounds like maybe somebody was required to give 30 days notice 
of service changes to certain customers with good laywers




This is my bet. Let's see. Peering went down October 6th. Then a fedexed 
letter with nasty threats arrives today [with a fax copy arriving on the 
6th]. Then a letter from Level3 fedexed back on the 7th for delivery on 
the 9th saying "We've cured the default and here is 30 days notice that 
we are terminating your account or changing your service."


Then 30 days from the 10th to November 9th...

Yup. Sounds about right.

Dave (I'm merging your comment in here too):

One question [referring to customers being used as pawns] -- what does 
ANY network have besides their customers to use in a peering negotiation?


Engineers often try to use things like prefix counts, # of routers, # of 
countries, blah blah blah but really these are all oblique references to 
the size and import of the single homed customers behind that network. 
Level3 thought their customers were more desirable than Cogent's. 
Level3's customers seem to have enough respect for Cogent's customers to 
make Level3 blink. I seem to remember the same thing happening with 
[another network] trying to depeer Cogent a few months back.


What I find interesting is that in this case, Cogent apparently allowed 
the sessions (whichever ones Level3 decided to fire back up) to come up 
cleanly. I think in that earlier instance Cogent kept the peering 
sessions down until some point [one speculates until the legal paperwork 
came through] -- notice I said "I think" I have no specific facts, just 
a vague recollection.


Taking this back to pricing (I suddenly found about 30 minutes I didn't 
expect to have), if Cogent terminated their most recent $6/mb/s or 
$8/mb/s promotion because of this concern with the Level3 peering and 
Level3 does indeed blink [comes back to the table and agrees to an 
indefinite term of SFI] Cogent might re-establish the promotion and with 
it grow by another 20 or 30 Gb/s or more.. possibly at the direct 
expense of Level3 and others and force more growth of their 
interconnections.


Another question (wow, 3 or 4 separate threads in one email). Instead of 
a "hard" de-peering. Wouldn't it be more effective if Level3 just 
stopped adding interconnection capacity to Cogent? That is an effective 
way of limiting the growth of losses between two networks, and if some 
discreet situation can be used [say a balanced traffic ratio] no new 
capacity will be added until that situation is resolved?  Cogent [in 
this case] could solve the problem by adding the appropriate # and type 
of customers or by buying transit to add capacity [and possibly 
forfeiting SFI]. Then instead of using your peering partner's customers 
as pawns, you are using your own to drive the business case either way.


That is a much "nicer" way to put pressure on a peer without it getting 
blasted all over the headlines... Especially if you are the network 
doing the depeering. And if your customer contracts don't cover 
(SLA-wise) interconnections, you don't care either way.


Just a few ideas...

Deepak

Re: Level 3's side of the story

[James]

On Fri, Oct 07, 2005 at 04:23:44PM -0700, william(at)elan.net wrote:
> 
> 
> And after November 9, what is then? Cogent/L3 depeering part 2?

Part III: Level3 Strikes Back :)

Depeering World Series now at:
 Level3: 2, Cogent: 14



-- 
James Jun
Infrastructure and Technology Services
TowardEX Technologies
Office +1-617-459-4051 x179 | Mobile +1-978-394-2867
[EMAIL PROTECTED] | www.towardex.com

Re: Level 3's side of the story

[william(at)elan.net]

BTW - it sounds like maybe somebody was required to give 30 days notice 
of service changes to certain customers with good laywers


On Fri, 7 Oct 2005, william(at)elan.net wrote:


And after November 9, what is then? Cogent/L3 depeering part 2?

On Fri, 7 Oct 2005, Richard A Steenbergen wrote:


http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-07-2005/0004164041&EDATE=

"On October 6, Level 3, as it had repeatedly advised Cogent it would,
terminated free traffic exchange with Cogent.  Because Internet users,
apparently without notice from Cogent and through no fault of their own,
have been impacted, Level 3 has, effective immediately, re-established a
free connection to Cogent.  In order to allow Internet users to make
alternative arrangements, we will maintain this connection until 6:00 a.m.
ET, November 9, 2005.  The effectiveness of this arrangement of course
depends on Cogent's willingness to maintain their side of the traffic
exchange.

RE: Level 3's side of the story

[David Hubbard]

From: William Allen Simpson
> 
> I don't remember seeing this public notice from Level(3) posted
> Wouldn't that be "without notice from Level(3)"?

They notified Cogent, not the public.  Cogent chose to
not do anything other than hope they won the staring
contest when Level 3 terminated the link, which
apparently they did.  Be interesting to see what
will happen in a month when they go at it again.

> 
> Splendid, that gives the world sufficient time to accept 
> Cogent's offer of 1 year free service.

This is not the first time Cogent has used their customers
as pawns in peering disputes, I don't know if I'd jump on
the bandwagon so quickly (spoken as a customer of both
companies).

David

Re: Level 3's side of the story

[william(at)elan.net]

And after November 9, what is then? Cogent/L3 depeering part 2?

On Fri, 7 Oct 2005, Richard A Steenbergen wrote:


http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-07-2005/0004164041&EDATE=

"On October 6, Level 3, as it had repeatedly advised Cogent it would,
terminated free traffic exchange with Cogent.  Because Internet users,
apparently without notice from Cogent and through no fault of their own,
have been impacted, Level 3 has, effective immediately, re-established a
free connection to Cogent.  In order to allow Internet users to make
alternative arrangements, we will maintain this connection until 6:00 a.m.
ET, November 9, 2005.  The effectiveness of this arrangement of course
depends on Cogent's willingness to maintain their side of the traffic
exchange.

Re: Level 3's side of the story

[William Allen Simpson]

Richard A Steenbergen wrote:

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-07-2005/0004164041&EDATE=

"On October 6, Level 3, as it had repeatedly advised Cogent it would, 
terminated free traffic exchange with Cogent.  Because Internet users, 
apparently without notice from Cogent and through no fault of their own, 


I don't remember seeing this public notice from Level(3) posted
Wouldn't that be "without notice from Level(3)"?


have been impacted, Level 3 has, effective immediately, re-established a 
free connection to Cogent.  In order to allow Internet users to make 
alternative arrangements, 


Splendid, that gives the world sufficient time to accept Cogent's offer
of 1 year free service.


we will maintain this connection until 6:00 a.m. 
ET, November 9, 2005.  The effectiveness of this arrangement of course 
depends on Cogent's willingness to maintain their side of the traffic 
exchange.



At which time Level(3) will have had time to purchase transit, as it
will be a "tier 2" hoisted on its own petard.

--
William Allen Simpson
Key fingerprint =  17 40 5E 67 15 6F 31 26  DD 0D B9 9B 6A 15 2C 32

Level 3's side of the story

[Richard A Steenbergen]

http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-07-2005/0004164041&EDATE=

"On October 6, Level 3, as it had repeatedly advised Cogent it would, 
terminated free traffic exchange with Cogent.  Because Internet users, 
apparently without notice from Cogent and through no fault of their own, 
have been impacted, Level 3 has, effective immediately, re-established a 
free connection to Cogent.  In order to allow Internet users to make 
alternative arrangements, we will maintain this connection until 6:00 a.m. 
ET, November 9, 2005.  The effectiveness of this arrangement of course 
depends on Cogent's willingness to maintain their side of the traffic 
exchange.

-- 
Richard A Steenbergen <[EMAIL PROTECTED]>   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)