NANOG 42 Peering BOF XVII and APRICOT 2008 Peering Forum
Hi all - In about a month (starting Feb 17 specifically) there will be two back-to-back events specifically for network engineers and peering coordinators; at NANOG 42, Feb 19, we will have the 17th Peering BOF (http://www.nanog.org/mtg-0802/agenda.html), and the following week, at APRICOT 2008, Feb 25-26 there will be an AP Peering Forum (http://www.apricot2008.net/P09.html#t1) held in Taipei, Taiwan. I need volunteer speakers and topics for both of these, and if you are a network operations person, I bet you might have a story to share with the group ! I'm looking for stories of interest to the peering community, such as: What processes did you use to transition an upgrade to your network? What worked well and what did not? What did you wish you knew when you built into and throughout different regions or countries? How did you select these locations (transport, transit, peering costs? attractive market? customers?) What was unexpected, or what lessons learned would you share with folks who would walk in your footprints? (technical, business, ops, legal, etc.) How did you decide which countries to build into, and what were the hidden costs and snafus? Basically, anything you would like to hear or would like to have heard would be appropriate. I have 45 minutes left to fill in the NANOG Peering BOF and a lot of time left for the APRICOT Peering Forum. If you can share your experiences on these or other topics at either or both of these forums, please send me a note ASAP. Also, we will once again have the Peering Personals at the Peering BOF and at the APRICOT Peering Forum, providing a chance for peering coordinators to introduce themselves to the group, describe their network and the types of networks they would like to peer with. If you would like to stand up and introduce yourself to the group, please fill out and send me the Peering Personals Form below. Thanks - William B. Norton for the Fabulous Internet Traveling Circus PS - Here is the URL to my working copy of the NANOG 42 Peering BOF Agenda - send me an email with additions, corrections, comments, etc. If you would like to edit the agenda that can be arranged as well: http://docs.google.com/Doc?docid=dddj3pds_41f5cwctng&hl=en Peering Personals Form - Peering Coordinator 1) Name: _ 2) email: _ Company: 3) AS#: __ 4) Peering Locations Now: __ 5) Peering Locations (Planned or future): _ 6) Transit traffic volume: ___ Mbps or Gbps 7) Traffic mostly outbound or inbound? __ 8) What are you looking for in a peer? __ (I'll ask this when you stand up to introduce yourself) 9) Why should folks want to peer with you? _ (I'll ask this also when you stand up) This information will appear on the screen behind you as you introduce yourself to the Peering BOF community. That way they can jot down whatever info they need and can put a face to a company, and hopefully have a discussion about peering at the break. We have found this to be a valuable way to facilitate the interaction between people who may not have an easier way to meet each other while at NANOG or APRICOT. I will allocate spots for 5 or so peering personals. /Peering Personal Form -- -- //-------- // William B. Norton // Co-Founder and Chief Technical Liaison, Equinix // Skype, Y!IM: williambnorton AIM wbnorton
Peering [EMAIL PROTECTED] BOF XV at NANOG 40 in Bellevue
[ If you are NOT ATTENDING this NANOG in Bellevue, please disregard
]
We have some time at this Peering BOF XV for some Peering Coordinator
introductions. This is
a chance for Peering Coordinators to introduce themselves to the group
before we break for beers.
How does this work? We solicit Peering Coordinators (with this note), asking
them to characterize
their networks and peering policies in general ways ("content heavy" or
"access (eyeball) -heavy,"
"Open" vs. "Selective" vs. "Restrictive" policies etc.).
From the answers we will select a set of ISP Peering Coordinators to share a
short (2-3 minute)
description of their network, what they look for in a peer, etc., allowing
the audience to put a face
with the name of the ISP. At the end of the Peering BOF, Peering
Coordinators will have time to
speak with Peering Coordinators of ISPs they seek to interconnect with. The
expectation is
that these interactions will lead to the Peering Negotiations stage, the
first step towards a more
fully meshed and therefore resilient Internet.
If you are a Peering Coordinator and wish to participate in the Peering
Personals section of the Peering BOF,
please reply to me (privately) with the answers the following 8 questions:
--
1) Name:
2) Title: ___
3) Company: _
4) AS#: _
5) Check which one best applies:
___ We are an ISP (sell access to the Internet)
--OR--
___ We are a Non-ISP (content company, etc.)
___ We are Content-Heavy
--OR--
___ We are Access-Heavy
6) Check whichever ones apply:
___ We are an "Open" peer (The answer to peering requests is YES)
___ We are a "Selective" peer (The answer to peering requests is YES
but we may have a few 'objective and meetable' pre-requisites such as
"three geographically diverse locations with 10Mbps on each coast")
___ We are a "Restrictive" peer (The answer to peering requests is generally NO)
___ We have huge volumes of traffic (lots of users and/or lots of content)
(huge: > 1 Gbps total outbound traffic to peers and transit providers)
___ We have a global network
___ We require Contracts for Peering
7) Current Peering Locations: ___
8) Planned (3-6 mos) Peering Locations: _______
Bill
--
//
// William B. Norton
// Co-Founder and Chief Technical Liaison, Equinix
// GSM Mobile: 650-315-8635
// Skype, Y!IM: williambnorton
Re: Internet Video: The Next Wave of Massive Disruption to the US Peering Ecosystem (v1.2)
Why are folks turning away 10G orders? I forgot to mention a couple other issues that folks brought up: 4) the 100G equipment won't be standardized for a few years yet, so folks will continue to trunk which presents its own challenges over time. 5) the last mile infrastructure may not be able to/willing to accept the competing video traffic . There was some disagreement among the group I discussed this point with however. A few of the cable operations guys said there is BW and the biz guys don't want to 'give it away' when there is a potential to charge or block (or rather mitigate the traffic as they do now). My favorite data point was from Geoff Huston who said that the cable companies are clinging to their 1998 business model as if it were relevent in the world where peer-2-peer for distribution of large objects has already won. He believes that the sophisticated peer-2-peer is encrypting and running over ports noone will shut off, the secure shell ports that are required for VPNs. So give up, be the best dumb pipes you can be I guess. Bill On 1/10/07, Brandon Butterworth <[EMAIL PROTECTED]> wrote: > Then that wouldn't be enough since the other Tier 1's would need to > upgrade their peering infrastructure to handle the larger peering > links (n*10G), having to argue to their CFO that they need to do it so > that their competitors can support the massive BW customers. Someone will take the business so that traffic is coming regardless, they can either be that peer or be the source with the cash. If they can't do either then they're not in business, I hope they wouldn't ignore it congesting their existing peers (I know...) > Then even if the peers all upgraded the peering gear at the same time, > the backbones would have to be upgraded as well to get that traffic > out of the IXes and out to the eyeball networks. The Internet doesn't scale, turn it off brandon -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Re: Internet Video: The Next Wave of Massive Disruption to the US Peering Ecosystem (v1.2)
On 1/9/07, Adam Rothschild <[EMAIL PROTECTED]> wrote: On 2007-01-09-12:08:16, "William B. Norton" <[EMAIL PROTECTED]> wrote: > [...] a few of the largest US ISPs are turning away these n*10G > Internet video transit customers ! I'd be interested in learning of specific vendors/markets, along with the reasons given. Did they cite temporary capacity constraints, or anything of greater long-term significance? Yea, I found that interesting as well. There were "cascading issues" cited by one Tier 1 ISP. First, the network equipment currently deployed hasn't been paid for and they would have to go back and argue for more $$ for a forklift upgrade. Which leads to the second reason - the colos are out of space/power or both. Usually both. So a forklift upgrade may be needed to replace the current gear with the new monster CRS or better class equipment to handle the emerging n*10G Video traffic demand. Then that wouldn't be enough since the other Tier 1's would need to upgrade their peering infrastructure to handle the larger peering links (n*10G), having to argue to their CFO that they need to do it so that their competitors can support the massive BW customers. Then even if the peers all upgraded the peering gear at the same time, the backbones would have to be upgraded as well to get that traffic out of the IXes and out to the eyeball networks. Bill Here in the New York metro, you'd be hard pressed to find a vendor willing to turn away a 10G transit deal and the associated revenue. In the past few months, I've been approached by half a dozen or so major carriers eager to sell 10 gigabit ports, and with the capacity to deliver. If your customers are, indeed, reporting a widespread difficulty obtaining 10 gigabit ports from the larger players, I can think of plenty of smaller ISPs and switch-based resellers who'd be happy to carry their traffic. While I'm greatly interested in Internet video and the need to come up with new ways to deliver it more efficiently, I'd be weary of listing the [lack of] availability of transit ports as contributing factor. -a -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Re: Internet Video: The Next Wave of Massive Disruption to the US Peer ing Ecosystem (v1.2)
On 1/9/07, Fergie <[EMAIL PROTECTED]> wrote: : Just as an observation, it appears to me (at least) that the most popular method of video distribution today is via GooTube. :-) I think it remains to be seen that that model will actually change dramatically to more of a "semi- real-time" model, regardless of the desires (or fears) of various vendors or operators. Hmm...I should have been more clear. I'm comparing the options a video guy has : buy transit to distribute the videos, buy CDN services, buy a mix or transit and peering, or use P2P. I have sample configurations and cost models for each, and cost them in units of $/video distributed for side to side comparison. From the reviews and discussions it was interesting how entrenched and enraged some people became when the p2p distribution model costed out to be the cheapest by far: Models A:10 videos/5 min B: 100/5min C: 1000/5min 1: Transit 1A: $0.60 1B: $0.36 1C: $0.20 2: CDN 2A: $0.77 2B: $0.44 2C: $0.24 3: Hybrid 3A: $0.69 3B: $0.31 3C: $0.17 4: P2P 4A:$0.18 4B: $0.0177 4C: $0.0018 Bill
Internet Video: The Next Wave of Massive Disruption to the US Peering Ecosystem (v1.2)
Hi all - Over the last year or so I have been working with Internet video companies who asked essentially the same question - "What is the most effective way of distributing massive quantities of Internet (video) traffic?" This has become a significant issue NOW because a few of the largest US ISPs are turning away these n*10G Internet video transit customers ! Thanks to all of you that shared your insights, or let me walk you through what this community has found to date, and especially those of you who shared their data points and allowed me to cite you as a source. I'm at the point now where I'd like to share the current draft (v1.2) of this discussion paper with a broader audience, epsecially those who will allow me to schedule a time to talk through the draft with you. (I have found this is the most effective way to get feedback next to face-to-face walkthroughs over lunch). Here's the Abstract: Video Internet: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem (v1.2) In previous research we documented three significant disruptions to the U.S. Peering Ecosystem as the Cable Companies, Large Scale Network Savvy Content Companies, and Tier 2 ISPs started peering openly. By peering with directly each other they effectively bypassed the Tier 1 ISPs resulting in improved performance, greater control over the end-user experience, and overall lower operating costs. This paper predicts a new wave of disruption that potentially dwarfs currently peered Internet traffic. Some of this emerging wave of Video Traffic is demonstrating viral properties, so the more popular videos are generating massive "Flash Crowd" effects. Viral Amplifiers (sites that do not host but rather highlight the most popular videos) amplify any viral properties a video may have. If we combine this flash crowd effect and the increased size of the video files downloaded, we see the crest of the first wave of significant incremental load on the Internet. The majority of this paper details four models for Internet Video Distribution (Transit, Content Delivery Networks, Transit/Peering/DIY CDN, Peer2Peer) across three load models. The cost models include network and server equipment along with pricing models for various distribution methods. Dozens of walkthroughs of this paper have led to stepwise refinement of the models and insights into why one would prefer or not prefer one model over the other. The summary is a comparison in cost-per-video across small, medium, and large distributions. The models (spreadsheets) can be made available to those interested. Bill -- //------------ // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Re: Internet 2010 - Predictions for 2010 from a Content Forum and NANOG 37 in San Jose
idn't get to do this at the Peering BOF at NANOG, but I did some table discussions outside in the hallways. There there was no voting so I am listing a subset of the predictions that seemed to resonate among a couple dozen or so folks at the hallway tables where question was discussed: "We are sitting around this table in 2010 at NANOG and we are commenting how remarkable the last few years have been, specifically that:" 1. We have 10G network interface(s) on laptops (I assumed wired, but someone else might have been thinking wireless) OK OK ! 10G wireless to the laptop is probably not plausible by 2010. 2. $5/mbps is the common/standard price of transit (other prediction was $30/mbps) Even the harh critics thought this was likely, the $5 transit price availibilty across the US 3. Internet traffic is now so heavily localized (as in 75% of telephone calls are across town type of thing but for the Internet) 4. Ad revenue will cover the cost/or subsidize significantly of DSL 5. 90% of Internet bits will be video traffic 6. VoIP traffic exceeds the PSTN traffic 7. Private networks predominantly migrate to overlays over the Internet 8. Wireless Internet Service Providers (WISPs) are serious competitive threat to DSL and Cable Internet 9. Sprint is bought by Time Warner 10. Cable companies form cabal & hookup with Sprint or Level 3 11. Government passes Net Neutrality Law of some flavor 12. Earthlink successfully reinvents themselves as Wireless Metro player in Response to ATT and Verizon 13. 40% paid or subscription as opposed to Content Click Ads. Like Cable Company channel packages, folks will flock to subscriptions for Internet Content packages. 14. RIAA proposes surcharge on network access (like Canada tax on blank CDs) 15. NetFlix conversion to Internet delivery of movies to Tivo or PC, or open source set top box 16. ISPs will be in pain Comment: "As opposed to now when we are living high on the hog?" 17. Last mile (fiber, wireless, …) in metro will be funded by municipal bonds 18. Death of TV ads, Death of broadcast TV, Tivo & Tivo like appliances all use the Internet with emergence of targeted ads based on demographic profiles of viewer 19. Google in charge of 20% of ALL ads (TV, Radio, Billboards, …) 20. Ubiquitous wifi in every metro with wifi roaming agreements The comments back said that this will certainly not be done by 2010 but maybe some initial movement towards this goal. 21. Congestion issues drive selective customer acceptance of partial transit offerings 22. IPTV fully embraced by cable cos – VOD – no need for VDR and ala carte video services replace analog frequency 23. Near simultaneous release of movies to the theaters, DVDs for the home, PPV, and Internet download to meet needs of different demographics. (Some get dressed up for theater, others have kids and can't leave home, others wantto watch on the flight to Tokyo – all watch the new release movie at about the same time) Video Peering For what it is worth, some of this resonates with the Peering BOF Video Peering discussion. With YouTube pushing 20Gbps after only one year in existance, and with the 30+ companies that often chase a high profile market such as theirs, we have a potential additional Internet load approaching 600Gbps! YouTube at the BOF said that their traffic is growing at about 20% per month, so it may be reasonable to expect their traffic to double a couple times over the next year. Even if you discount the competitors traffic flows, video still appears as a *massive* traffic volume coming into the Peering Ecosystem over the next bunch of months. There has been some dicussion on various IRC channels asking "why is video peering" separately noted from regular old "Content" being peered. I tend to differentiate it solely because the volume of this traffic is so large and expected to grow as codecs march to convey HDTV quality video over the net, either on demand or streamed to a box for buffered delivery or downloaded. The point is the traffic volume for this type of peered traffic is so large and it will grow over time, which makes it interesting to watch from a peering research perspective. And yes, they are willing to peer the traffic for free so you eyeball networks and they (YouTube) don't have to pay transit fees on the traffic. Bill -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Internet 2010 - Predictions for 2010 from a Content Forum and NANOG 37 in San Jose
of movies to the theaters, DVDs for the home, PPV, and Internet download to meet needs of different demographics. (Some get dressed up for theater, others have kids and can't leave home, others wantto watch on the flight to Tokyo – all watch the new release movie at about the same time) Video Peering For what it is worth, some of this resonates with the Peering BOF Video Peering discussion. With YouTube pushing 20Gbps after only one year in existance, and with the 30+ companies that often chase a high profile market such as theirs, we have a potential additional Internet load approaching 600Gbps! YouTube at the BOF said that their traffic is growing at about 20% per month, so it may be reasonable to expect their traffic to double a couple times over the next year. Even if you discount the competitors traffic flows, video still appears as a *massive* traffic volume coming into the Peering Ecosystem over the next bunch of months. And yes, they are willing to peer the traffic for free so you eyeball networks and they (YouTube) don't have to pay transit fees on the traffic. Bill -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Peering BOF XI Meeting Minutes ----- D R A F T
Peering BOF XI Meeting Minutes Facilitator: William B. Norton <[EMAIL PROTECTED]> Agenda 1. Notes from the Field - wbn 2. ad Hoc Transit Survey – Dave Wodelet 3. Paid Peering as an Adjunct to Settlement Free Interconnect (15 min). -wbn 4. The Great Peering Debate (30 min) ras, ianai 5. Peering Contact Database (peeringdb.com) Update (10 min). - ras 6. AS7018 & AS7132 Discussion (15 min) 7. Peering Personals – all (remainder) 1) Notes from the Field Trying something new this NANOG Peering BOF: Community Seating This was a hit! We had 200 chairs arranged in three concentric ovals so everyone could see everyone else. The protocol for speaking did not involve queueing up at a microphone but rather standing up and speaking your peace. It took a little while to get used to it but it certainly facilitated very interactive and lively community discussions. Peering Ecosystems Update – Italy Peering Ecosystem Exception Telecom Italia peers its domestic routes openly in a single (or if needed two) location in Italy. This is counter to every peering ecosystem I have studied; Tier 1 ISPs (those with access to the entire routing table for the country solely via peering relationships) typically will not peer with anyone else, making Telecom Italia and Italy an interesting exception. Daniele (from NaMex) pointed out the recently privatized Telecom Italia peered openly to avoid government regulation, and I mentioned that they also pointed to increased revenue from fiber and copper sales to successful competitive ISPs. Note that this is a different entity from Telecom Italia Sparkle, which is the sole international transit provider for TI and does not peer openly. Job (from AMS-IX) mentioned that KPN is similarly openly peering. - Past Peering Debate documented The last NANOG Peering Ratios debate spilled over into heated discussions in the hallways and I went ahead and documented the strongest arguments overheard on both sides of the issue. I have this white paper available to anyone who wishes to better understand the Peering Ratios issue. This white paper is now freely available. Send email to [EMAIL PROTECTED] with "The Folly of Peering Ratios?" in the subject line and I'll be glad to send a copy of the "The Folly of Peering Ratios?" white paper. 2) Ad Hoc Transit Survey – Dave Wodelet ([EMAIL PROTECTED]) Dave was interested in continuing the transit survey we have done in the past peering bofs with a little more specificity, so he asked several questions: a) who are your upstream transit providers? b) what are the prices per mbps and commits? c) how old is the contract? d) are there any special aspects of the deal? Dave agreed to report back on the results at the next NANOG Peering BOF. 3) Top 8 reasons Paid Peering Has Not Taken Off The Peering community has discussed paid peering for over a decade. The question was floated around "Why hasn't paid peering taken off?" resulting in the following answers: 1. Unpredictability & Budgeting & Gaming the system 2. We are "peers" & we shouldn't pay. 3. If we pay now it will be harder to migrate to settlement free later 4. No Signatory Authority for $$$ - another group handles vendors/paying 5. Transit Price < Paid Peering Price – why bother, use communities to limit transit, have a backup just in case 6. We don't have that product to offer 7. Low margin product – the margins are already thin on transit, too much effort 8. I don't want to screw up my existing peering relationships. Even if I get some $$, I piss off a peer I care about. This was also a lively discussion resulting in a total of eight reasons when my hallway conversations initially resulted in five reasons why paid peering hasn't taken off. 4) "The Utility of MEDs" Peering Debate • Peering Question: "From a Practical Perspective, are MEDs useful for distributing the peering traffic load?" 2 minutes each: 1. "MEDs are not a useful tool for Peering" - ras 2. "MEDs are a useful tool for Peering" - patrick 3. Ras attacks "MEDs are a useful tool for Peering" position 4. Patrick attacks "MEDs are not a useful tool for Peering" position 5. Closing Arguments • Initial vote: How many people believe MEDs are a useful tool for distributing peering traffic? Group Answer by Vote: MEDs ARE useful. Debate begins... "MEDs are not a useful tool for Peering" - ras 1. MEDs worsen route quality and increase latency 2. Costs go up overall – just shifting costs around 3. Foolhardy to assume you can route to someone else's network better than they can "MEDs are a useful tool for Peering" - patrick 1. Judicious use of MEDs can improve performance 2. MEDs can remove the technical objections to peering 3. It's foolhardy to assume other people can route properly inside their own network. • Audience votes "Which side made the more
The Folly of Peering Ratios
Hi all - At the Peering BOF X at NANOG 35 in Los Angeles last week we held a debate on the rationality of Peering Traffic Ratios as a Peering Partner selection criteria. During the debate both sides made good points, but interestingly, some of the strongest arguments on both sides of the debate came out during the Q&A and during coffee break/bar/beer-n-gear ad hoc debates that followed. I have classified roughly six arguments culled from these discussion, attributed where I remembered the source, along with the corresponding counter arguments that seemed to collectively reveal the folly of peering traffic ratios as a peering discriminator. I have documented and diagrammed these arguments in the form of a white paper titled, of course, "The Folly of Peering Traffic Ratios." I am looking for some people to review the paper, provide feedback, better defend an argument on either side, provide anecdotes, whatever you believe would help make the paper an accurate portrayal of the arguments on both sides of this issue. If you have the time to let me walk you through the paper over the phone (about 20 minutes) and discuss, that would be the best - I find I get the most feedback this way. I'll send out a note to the list when I have done enough walk throughs to feel comfortable enough that I have things about right and that the draft can be circulated more broadly. Here is the current summary from The Folly of Peering Ratios v0.5: -- snip --- Summary In the heated discussions surrounding this topic there were six flavors of arguments put forth to support peering traffic ratios as a peering selection criteria. Argument #1 - "I don't want to haul your content all over the world for free." This argument is countered with the observation that, whether peered with a content heavy or an access heavy ISP, the ISP is being paid by its customers for providing access to that traffic. It is not hauling the traffic for free. Argument #2 – "OK, but there is massive asymmetry here. Look at how many bits miles I have to carry your content, while you have only to deliver your content across the exchange point." Regardless of whether peered with content or access the load on your network is about the same; the access or content traffic has to traverse your network to get to your customers. This is an argument perhaps for more geographically distributed points of interconnection. Argument #3 – "As an Access Heavy ISP, I don't want to peer with Content Heavy ISPs, because doing so will screw up my peering traffic ratios with my other peers." The analysis and diagrammed traffic flows in this paper prove this assertion true, however, the argument is circular: 'Peering Traffic Ratios are valid criteria because they support Peering Traffic Ratios with others.' Argument #4 – "I want revenue for carrying your packets." While this argument is business rational, peering traffic ratios are a poor indicator of the value derived from the interconnection. Argument #5 – "My backbone is heavily loaded in one direction – I don't have $ to upgrade the congested part of my core without a corresponding increase in revenue." This loading problem can better be solved with better traffic engineering, with more resources for the core or by temporarily postponing all additional peering until the core is upgraded. Introducing a peering traffic ratio requirement is at best a temporary solution and signals a poorly managed network – a bad image for peers and transit customers alike. Argument #6 – "I don't want to peer with anyone else. Peering ratios help me systematically keep people out." This is an understandable (although seldom articulated out loud) argument but is a poor defense for peering traffic ratios per se since it is an arbitrary discriminator; one could just as well have specified the size of the backbone core, the market capitalization or debt structure, or an extremely large number of distributed interconnect points. Peering Traffic Ratios emerged years ago in the Internet as a peering candidate discriminator, but appear to have their roots in the PSTN world. Ratios reflect the telephony settlement model of buying and selling minutes, where the traffic difference between in and out is used for settling at the end of the month. However, the Peering Traffic Ratio discriminator model for the Internet interconnection fails because traffic ratios are a poor indicator of relative value derived from the interconnect. If you have time to review the paper (about 6 pages), please send me a note. Thanks! Bill -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Peering BOF X Detailed Agenda
Hi all - I'm happy to announce the 10th NANOG Peering BOF will be held in Los Angeles from 14:00 to 15:30 on Monday. We have a *very* full Peering BOF agenda: 14:00 Introduction / Welcome - William B. Norton (Equinix) 14:05 Ad Hoc Transit Survey - all Since the Peering vs. Transit issue contains a financial component, we will employ an anonymous straw poll of wholesale transit prices and do a quick Peering Breakeven Point calculation by the end of the BOF. 14:10 Peeringdb.com - Richard Steenbergen (nLayer) The peeringdb attempts to solve two related Peering Coordinator Community problems - first, one of the most difficult problems is finding out who to talk to about peering in the target ISP company. Second, IXes have had a very difficult time populating and maintaining peering contact information for its populations. Richard has volunteered and set up a central database of peering contact information for the Peering Coordinator Community and he will share some stats and encourage folks to register their peering information. 14:15 Good Peers / Good Customers - Peter Cohen (Telia) Are all peers created equal? Should an ISP prefer some customers over others based on their traffic patterns? Peter will share some insights into these questions. 14:30 Unified Peering Forum - Josh Snowhorn (Terramark) Josh will share updates regarding the combined peering forum initiative, intended to reduce Peering Coordinator travel expenses and maximize the chances for peering coordinators to meet each other. In the last few years, the number of peering forums grew, and the Peering Coordinator Community attending these various forums to some degree fractured, reducing the benefits to the community. This initiative seeks to, among other things, pull together the community to fewer, open and jointly run forums. 14:40 The Great Debate on Peering Ratios : Important Metric or Dinosaur PreReq of a Bygone Era - Peter Cohen vs. Richard Steenbergen One challenge the Peering Coordinator Community faces (besides making contact and working within financial constraints discussed earlier) is meeting the peering requirements of the large traffic potential peers. There are some peers that believe peering ratios help them scrutinize where to most fairly expend their engineering resources, and ratios are one way to determine the balance of benefits between the potential peers. The other view is that there is no valid technical reason to use ratios are a filter for potential peers. A: Opening Statement - Peering Ratios are a valuable metric B: Opening Statement - Peering Ratios have no technical merit for screening potential peers A: Attack B / Defend A position B: Attack A / Defend B position A: Closing Statement B: Closing Statement Audience Vote: Which side made the more compelling case? Audience Engages Debaters: A chance for the audience to ask questions and make points NOT made during the debate, or to help reinforce points not made strongly enough during the debate. Secondary Audience Vote - did the audience view change? With the advantage of the audience points and followup discussion, which side made the more compelling case? 15:20 Peering Personals - all We have a few minutes for those who travelled a great distance to introduce themselves to the Peering Coordinator Community as we break for coffee and cookies. If you would like to introduce yourself to the Peering Coordinator Community, please send a note to [EMAIL PROTECTED] BEFORE MONDAY including: Name: email: Company: AS#: Three things the Peering Coordinator Community should know about peering with you: 1) 2) 3) These three things will help me select who gets stage time in case there are more people than time, and please make sure you speak with me before the BOF so we can seat you up front in the interests of time.
Re: Cogent/Level 3 depeering (philosophical solution)
Peering Ratios?
It is very timely that the upcoming NANOG Peering BOF X in Los Angeles
will have a debate on this very subject: Traffic Ratios - a valid
settlement metric or dinosaur from the dot.bomb past.
I'm sure the strongest arguments from these threads will be clearly
articulated (in a bullet point/summarized form I hope) during the
debate by the debaters. At the end of the day, as with most things
peering, the focus of this discussion is a meld of business and
technical interests. The heat we have witnessed is probably more
related to the friction of the business interests. We get very upset
about the notion of "fair" don't we. Perhaps in the few structured
minutes of the Peering BOF debate we can objectively hear both sides
of this argument and provide a little light as well.
Defending Traffic Ratios as a valid peering prereq: Peter Cohen
Attacking Traffic Ratios as peering prereq: Richard Steenbergen
Should be good fun.
Bill
On 10/10/05, David Schwartz <[EMAIL PROTECTED]> wrote:
>
>
> > [EMAIL PROTECTED] ("David Schwartz") writes:
>
> > > I think the industry simply needs to accept that it's more
> > > expensive to receive traffic than to send it.
>
> > It is? For everybody? For always? That's a BIG statement. Can
> > you justify?
>
>In those cases where it in fact is and there's nothing you can do
> about it,
> you need to accept it. You should not expect to be able to shift the burden
> of carrying your customers' traffic on your network to others. (The fact
> that you can sometimes bully or blackmail and get away with it doesn't
> justify it.)
>
> > > ...
> > > The question is whether the benefit to each side exceeds their cost.
>
> > Yea, verily. But I don't think you'll find a one-cost-fits-all
> > model. When
> > one person's costs are lower than another and they're doing
> > similar things,
> > it's often called "efficiency" or "competitiveness". (Just as
> > one example.)
>
>I heartily agree.
>
>My point is simply that the "your customers are getting more out of our
> network that our customers are" argument is bull. Your customers are paying
> you to carry their traffic over your network.
>
>There can certainly be legitimate peering disputes about where to peer
> and
> whether there are enough peering points. If someone wants you to peer with
> them at just one place, it would certainly be more cost-effective for you to
> reach them through a transit provider you meet in multiple places, for
> example. (You could definitely refuse settlement-free peering if it actually
> increases your costs to reach the peer.)
>
>I am not making the pie-in-the-sky argument that everyone should peer
> with
> everyone else. I am specifically rejecting the argument that a traffic
> direction imbalance is grounds for rejecting settlement-free peering. If
> your customers want to receive traffic and receiving is more expensive, then
> that's what they're paying you for.
>
>Again, carrying *your* customers' traffic over *your* network is what
> *your* customers are paying *you* for. If your customers want more expensive
> traffic, you should bear that greater burden.
>
>A traffic direction imbalance is not reasonable grounds for rejecting
> SFI.
> The direction your customers want their traffic to go is more valuable and
> it's okay if it costs more.
>
>DS
>
>
>
--
//
// William B. Norton <[EMAIL PROTECTED]>
// Co-Founder and Chief Technical Liaison, Equinix
// GSM Mobile: 650-315-8635
// Skype, Y!IM: williambnorton
Peering BOF IX Meeting Minutes
ersonals – (jotted down everything on the sheets) -- Korea Telecom – 4766 – Bong-Hwa Song – 40 Gbps Capacity, peering in Seattle PAIX, Palo Alto PAIX, EQLA, LINX, AMSIX, [EMAIL PROTECTED] AARNET – 7575 – Mark Prior – Research Net, Seattle SIX, PAIX, LAIIX, LAAP, FixW, P|Wave, [EMAIL PROTECTED] Netflix – Vish Y – [EMAIL PROTECTED] – Selective peer Steve Gibbard – PCH – 42, 3856 – Data Collection – Open Peering 0 25 locations all over the world (SIX, EQLA, LAIIX, NYIIX, PAIXNY, EQASH,NOTA,…) Steve Gibbard – Hurricane Electric – 6939 – [EMAIL PROTECTED], [EMAIL PROTECTED] – 2700 routes, public but migrating to private at EQSJO, EQLA, EQCHI, EQDAL, EQASH, PAIXPA,NYIIX, LINX, AMSIX Brokaw Price – Yahoo! – 10310 – OPEN – [EMAIL PROTECTED] Todd Underwood – 64597 – [EMAIL PROTECTED] – NOTA, LINX, Goofy peering, route collection only, store updates, Google – Maurice Dean – 15169 – selective peering – [EMAIL PROTECTED], NYIIX, EQ-CHI, EQCHI, EQASH, NOTA, PAIXPA,PAIXVA,1118th, TelX (60 Hudson)., LINX< AMSIX, LAIIX Akamai – 1 – Patrick Gilmore – 20940 outside U.S., OPEN and at every IX, content heavy, [EMAIL PROTECTED] ESNET – 293 – Yvonne – ipv4, ipv6, multicast, EQASH, EQSJO, MAEEast MAEWest, PAIX, AADS, FixW, Pacific NW Gigapop, [EMAIL PROTECTED], selective peering policy with AUP Celeste Anderson – AS4, AS27, AS2152/2153, ISI/Los Nettos, research net, lots of eyeballs, LAIIX, LAAP, Pacific Wave, PAIX LA, Mostly OPEN~selective, eyeballs, [EMAIL PROTECTED], [EMAIL PROTECTED], Matt Peterson – SixApart , AS# pending, SIX, EQSJO, LiveJournal.com, [EMAIL PROTECTED] -- Hope this help! -- // // William B. Norton <[EMAIL PROTECTED]> // Co-Founder and Chief Technical Liaison, Equinix, Inc. // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Peering BOF IX at NANOG in Seattle - The Great Public vs. Private Peering Debate
Hi all - Just wanted to invite you all to the upcoming Peering Birds-Of-a-Feather session at the upcoming NANOG, and give you a flavor of a couple of the topics to be discussed... Peering Introductions --- For Peering Coordinators who would lke to introduce themselves to the North American Peering Coordinator Community, we have some time set aside for you to introduce yourself and share with the Community: Company Name and Contact Information, AS#, Where you are peering or expect to peer in the future, What you are looking for in a peer, and Why people should be interested in peering with you. We have found these face-to-face interactions helps facilitate peering, particularly for folks coming in from overseas. It helps to bring network maps, and lots of business cards. If you email [EMAIL PROTECTED] this information I will make a slide with your contact information on it that will show up behind you are you speak. The Public vs. Private Peering Debate We have recruited two Peering Coordinators to articulate the two sides of the Great (Public vs. Private) Peering Debate. They have graciously agreed to share with the audience the Strongest Arguments for Public Peering (Maurice Dean), and the Strongest Arguments for Private Peering (Peter Cohen). Each side will have a few minutes to present their case, then a few minutes to attack the claims made by the other side and/or reinforce their own side of the argument as needed. We will perhaps add a few minutes in the middle for a couple of limited scope audience questions; those to help the speakers clarify a point (no speeches or attacks here). Both sides will then summarize their argument and the audience will be asked to vote for which side made the more compelling case. Audience Discussion --- As we did last Peering BOF, we will open the floor to discussion, focusing on points that one or both sides failed to make, or failed to make strongly enough, that would have perhaps made a difference in the audience vote. Background -- I researched this issue with a subset of the Peering Coordinator Community and shared the early results at the RIPE EIX WG meeting. If the discussions there are any indicator, I think we are in for an interesting and educational community discussion here. Below is an excerpt of the public vs. private peering arguments I heard from the Peering Coordinator Community and shared at the RIPE EIX WG meeting. I agree with the Peering Coordinators who believe the answer for most ISPs is a hybrid of public and private peering. I also agree that perhaps there sometimes emerges a transition based upon scale and strategic intent, but we will see what the community comes up with at the BOF. Bill PS - I cut and pasted the text below from the "The Great (Public vs. Private Peering) Debate: Peering at 10G" white paper that I am using to document these debates as they relate to 10 Gigabit-per-second Ethernet Peering. I am still looking for reviewers to provide feedback here BTW...If you are interested in this stuff and can spend a little time to provide feedback, send email to [EMAIL PROTECTED] When I feel more comfortable that I have it right, I will make the paper freely available to anyone who would like a copy. --- : : The Top 4 Reasons Public Peering is better than Private Peering 1. Aggregation Benefits a. A network can easily aggregate a large number of relatively small peering sessions across a single fixed-cost peering port, with zero incremental cost per peer. (Private peering requires additional cross connects and potentially an additional interface card, so there are real costs associated with each incremental peering session.) Small peering sessions often exhibit a high degree of variability in their traffic levels, making them perfect for aggregation. Since not all peers peak at the same time, multiple peers can be multiplexed onto the shared peering fabric, with one peers peak traffic filling in the valleys of another peer's traffic. This helps make peering very cost effective: "I can't afford to dedicate a whole gigE card to private peering with this guy, but public peering is a no-brainer." b. Public peering ports usually have very large gradations of bandwidth: 100Mbps Ethernet upgrades to 1Gbps Ethernet, which upgrades to 10Gbps Ethernet. With such large gradations, it is easier for smaller peers to maintain several times more capacity via public peering than they are currently using, which reduces the likelihood of congestion due to shifting traffic patterns, bursty traffic, or uncontrolled Denial of Service attacks. "Some peers aren't as responsive to upgrading their peering infrastructure, nor are they of similar mind with respect for the desire for peering bandwidth headroom[1]." The
NANOG 33 Peering BOF VIII Notes
ees. Maybe you shouldn't be peering then. +The ability to try an IX before you buy without going through a difficult install is a strong benefit of remote peering. +Lack of visbility is lessened when Remote Peering is used for Private Peering +It is a fallacy that "It just feels wrong" - The world is complex â deal with it! +The guys running these MPLS cloud are doing a great job; you may seen some latency/jitter during regrooming but that is about it. Not that instable. +Saves money, both capex and opex - Creating a "Fake" network to meet peering prereqs is not generating an ideal Internet topology. - May save $ but Peering Policies will soon adjust to deal with this, making it difficult for "real" networks to get peering. - General â More Complexity->More Problems->BGP instability for all of us Community VOTE: We took a different vote, given the points raised by the broader audience along with the debater points, asking the question: Is remote peering a good thing? 33 Is remote peering a bad thing? 24 Interesting â I would have thought the peering coordinators would have said uniformly that remote peering was a bad thing, but given all the points shared and discussed, the Peering Community in the room were clearly more accepting of remote peering. Peeringdb.com â Richard Steenbergen Richard shared his community service project to collect and disseminate Peering Contacts in the Peering Coordinator Community. The group felt this was a valuable and noble endeavor and thanks Richard for taking this on. -- Peering Personals Yahoo! AS#: 10310 US, 15635 in the UK Brokaw Price [EMAIL PROTECTED] Samsung Networks AS#: 6619 Woohyong Choi <[EMAIL PROTECTED]> McLeodUSA Telecommunications AS#: 7228 Thomas Barrera [EMAIL PROTECTED] VitalStream, Inc. AS#:13980, 30282, 30636, 30637 Kim W. Summers [EMAIL PROTECTED] NASA UNITeS / SAIC AS#: 297 Michael Turner [EMAIL PROTECTED] USC AS#: 226 Walt Prue [EMAIL PROTECTED] CENIC AS#: 2152 Darrell Newcomb [EMAIL PROTECTED] Company: Akamai Technologies AS#: 1, 20940 âPatrick W. Gilmore [EMAIL PROTECTED] Title: Director, Network Strategy & Support Webair Internet Development Inc AS#:27257 Sagi Brody [EMAIL PROTECTED] TELUS AS#: 852 Domenica Roda / Geoffrey Holan <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> Net Access Corp. AS#: 8001 Steven J. Schecter <[EMAIL PROTECTED]> Giganews AS#: 30094 Edward Henigin [EMAIL PROTECTED] University of Washington / Pacific Northwest Gigapop AS#: 73 / 101 Dave McGaugh [EMAIL PROTECTED] Electric Lightwave, LLC AS#: 5650 Steven Raymond <[EMAIL PROTECTED]> China Telecom (USA) AS#: 4134 Joe Zhu <[EMAIL PROTECTED]> ESnet AS#293 Name:Yvonne Hines [EMAIL PROTECTED] -- Closing â The Peering BOF closed around 11:00 but people hung around and talked toward midnight. This time slot might be a bit tough for those on Eastern Time who were effectively up until 3AM discussing peering, and rehashing the great debate. -- // // William B. Norton <[EMAIL PROTECTED]> // GSM Mobile: 650-315-8635
NANOG 33 Peering BOF VIII
Peering Coordinators attending NANOG in Las Vegas - We have a very exciting (and very full) Peering BOF agenda...Let me give you a flavor for what we are doing this year. At 9PM we'll start out with a "State of the Peering Internet" and, with the audience, identify a few key trends and the most pressing issues of the Peering Coordinator Community. Next we repeat last year's highly successful "Great Debate", this time on the topic of "Remote Peering : the good vs. the bad and the ugly". Remote Peering is peering without a local physical presence : tunneling ethernet frames across the ocean over MPLS into a peering fabric for example. Stephen Wilcox has graciously stepped up to present the con position - the bad and the ugly of remote peering. I have a couple volunteers to present the pro position, but I would prefer to have a Peering Coordinator that is actually using remote peering to advocate its use. If noone volunteers I have two backup debaters that have agreed to present the pro side of the debate and to debate Stephen on this matter. At the end, the audience will vote by show of hands for who made the more compelling case. If anyone would like to volunteer a prize for the winner please let me know ;-) Next, Richard Steenbergen will briefly share his Peering Coordinator community work with peeringdb.com, a resource for Peering Coordinators to obtain and share contact information. We will finish off with the Peering Personals, a chance for Peering Coordinators to introduce themselves to the group, talk about their network, what they are looking for in a peer, where they peer, their AS# and contact info, etc. I will use the RSVP form below to make sure that the relevant information is ready for Peering Coordinators in the audience to jot down in case they don't meet up with the speaker. We have found in the past that Peering Personals is very effective in facilitating peering. If you are a Peering Coordinator and would like to participate in Peering Personals, it is a great way for others to put a face to a network and AS, and to meet the North American Peering Coordinator Community. Please fill out the following form and e-mail to me ([EMAIL PROTECTED]). We only have time for about 10 Peering Coordinators, so please respond asap: Name: Email: Title: Company: AS#: Check each that applies: ___ We are an ISP (sell access to the Internet) --OR-- ___ We are a Non-ISP (content company, etc.) ___ We are Content-Heavy --OR-- ___ We are Access-Heavy ___ Peering with Content Players or Content Heavy ISPs is OK by us ___ We generally require peering in multiple locations OR ___ We will peer with anyone in any single location ___ We have huge volumes of traffic (lots of users and/or lots of content) (huge: > 1 Gbps total outbound traffic to peers and transit providers) ___ We have a global network ___ We require Contracts for Peering Current Peering Locations: ___ Planned (3-6 mos) Peering Locations: ___ After the BOF we will break for beers and exchange of business cards. Bill
RE: Current street prices for US Internet Transit
First - As for whether the US Transit market is healthy or unhealthy... I am not privy to the ISP calculations that demonstrate financial viability at these prices, so I can only go on the sentiments expressed by folks that have done the analysis for their companies and have shared their views with me or my colleagues. Having said that...It certainly appears that the majors can't make money in this transit market, and they would probably say that the bottom feeders are greasing the skids to financial ruin for this industry. At 08:50 PM 8/16/2004 -0700, Michel Py wrote: The really interesting question IMHO is this: does said content company also peers, or just buys transit? I read in Bill Norton's papers that a content company is a good candidate to peer with large tier-2s. Well, it is the *large* network-savvy Content Guys that I see peering quite a bit of their traffic. For these guys the motivation seems to be the performance benefits (it is all about pages coming up fast, end user behavior stuff), but the financial savings isn't far behind. FOr these guys the savings are still in the millions per annum. And these are the guys that do enough volume to negotiate transit prices in the teens. > The Cost of Internet Transit in.. > Commit AU SG JP HK USA > 1 Mbps $720$625$490$185$125 > 10 Mbps $410$350$150$100$80 > 100 Mbps$325$210$110$80 $45 > 1000 Mbps $305$115$50 $50 $30 Someone mentioned that this was comparing apples to oranges. Indeed it is, I would disagree that these are apples and oranges comparisons - these are real prices (normalized to USD) for transit that someone in the country would pay to access the big 'I' Internet. The Peering Coordinators I spoke with that have expanded into these markets did point out that each Peering Ecosystem differs - I think I documented about 10 differences in the Asia Pacific Peering Guidebook - but, ultimately, they will need to buy transit in that market. So these numbers are useful for budgeting for network expansion into Asia. I would also share that the local loop prices in these other markets vary widely ; an OC-3 local loop in Singapore may be twice the cost of a distance insensitive OC-3 local loop in Hong Kong. To that end, the business cases for peering will be quite different. Bill
Re: Current street prices for US Internet Transit
Patrick -
The other thing that I found interesting when factoring in the equipment
costs into the cost of Peering, was that the used equipment market remains
vibrant.
From my conversations with folks in the Peering Coordinator Community,
round numbers here, one can pick up a used 7500 series router equipment now
for about $9K ! The configuration was with an OC-3, and FastE for peering,
for about 25% of the new cost. Pretty amazing - from my research I cited
one guy who built his whole test lab from used 7500's. On eBay folks can
get used Juniper M20s as well, some still shrink wrapped. Caveat: When I
walked the Cisco and Juniper contacts through the research paper ("Do
ATM-based Internet Exchanges Make Sense Anymore?") they pointed to the
software license as being non-transferable, and therefore requiring a new
license from the vendor to be legitimate. There is also a re-certification
process if you want to get the gear under service contract. There are some
used equipment vendors that claim to take care of these issues for you.
So, used equipment is one way that some are deploying low cost networks,
and yes, the packets get there. If their negotiating is as strong as their
scrounging, they may be able to compete in today's market.
Bill
RE: Current street prices for US Internet Transit
Thanks to all who replied with data, and yes, the pricing was all 95th percentile. Wow - the U.S. has an amazingly unhealthy and cut throat transit market in 2004. About 20 folks responded, most saying the Peering Coordinator quotes (below) sounded about right. > ISP Transit Commits and Prices - > if you commit to1M per month you will pay about $125/Mbps > if you commit to 10M per month you will pay about $ 60/Mbps > if you commit to 100M per month you will pay about $45/Mbps > if you commit to 1000M per month you will pay about $30/Mbps A couple people said these prices were TOO HIGH, particularly for the gig commit, although several multi-gig commits came in tiered; for example, $45/Mbps for 1G commit, $35 for 2G, etc. on down to $21 for 8G commit. (One Tier 2 ISP said that they sold 1G commit as low as $18/Mbps, presumably simply reselling Tier 1 BW so the difference may be negligible.) Three said that these transit prices were TOO LOW, in one case they paid about double these numbers. It was interesting that these three were a content company, a cable company and a DSL company, folks who traditionally don't sell transit. Maybe they are in a retail market for transit, while everyone else buys in the wholesale market. Since so many said these prices are about right, I'll use them for the Peering versus Transit analysis. A couple people pointed to the 10M commit being closer to $80/Mbps, so that may be an adjustment. Given the adjustment, I thought you might be interested in how the U.S. transit prices compare against a handful of other Peering Ecosystems: The Cost of Internet Transit in Commit AU SG JP HK USA 1 Mbps $720$625$490$185$125 10 Mbps $410$350$150$100$80 100 Mbps$325$210$110$80 $45 1000 Mbps $305$115$50 $50 $30 Round numbers anyway FWIW. Hope this helps. I feel bad for those selling transit these days - at these prices, margins must be mighty thin, and I suspect we will see some more turbulence in the industry. Bill
Current street prices for US Internet Transit
Hi all - As I'm putting together some Peering vs. Transit pricing models for an upcoming peering forum in NYC, I'd like to verify some current transit prices I am getting from the field. Here is what the peering coordinators are telling me... Sound about right to you guys? ISP Transit Commits and Prices (Tier 2 ISP buying from non-bottom feeder ISPs) - if you commit to1M per month you will pay about $125/Mbps if you commit to 10M per month you will pay about $ 60/Mbps if you commit to 100M per month you will pay about $45/Mbps if you commit to 1000M per month you will pay about $30/Mbps Round Number Costs for Local Loops into an IX 100M $1000/month 1000M $4000/month If you have any actual price quote #'s that I can generalize that would be great. If you can say too high, too low, about right based on the commits, that would help also. Please note that I'm not looking for the best price one could get, but rather an average street price a Tier 2 ISP or cable company (or network savvy Content company) would pay. These prices are going to used for modelling The Business Case for Peering v2, an update to the original white paper describing when peering makes sense financially... more on this later. Thanks! Bill
Asia Pacific Peering Guidebook Reviewer Request
Hi all - If you have no interest in Internet Peering in Asia, read no further. Otherwise,... Over the last year I have been working with the Peering Coordinator Community, particularly those that have built into and throughout Asia, to document what is *different* about peering in Asia as compared to say the U.S. and Europe. In February of this year I pulled together a group of Peering Coordinators for the APRICOT Peering Track. I asked them to share Peering processes that they found different, interesting, counter intuitive, or unexpected. This white paper documents the lessons learned and insights gained from Peering Coordinators experienced with peering in Asia. The Asia Pacific Peering Guidebook follows the path set by "The Evolution of the US Peering Ecosystem" white paper that I presented at the last NANOG, but focuses on the common dynamics and experiences that folks shared with me over the last year and at the APRICOT Peering Track. First, the paper documents what I call the "Foreign Tier 1 Peering Dynamic" that casts a Tier 1 ISP in one Peering Ecosystem as a Tier 2 ISP in another. We focus on the motivations, the reasons why this occurs as ISPs expand globally. Next, Peering Coordinators shared five reasons that they expanded into/within Asia: 1) Incumbent Tier 1 ISPs to peer their routes outside their home market 2) Meet US Tier 1 ISP Peering Prerequisites 3) Customers want them in Asia 4) Global Marketing Benefits 5) Sell transit in a more attractive transit market using four methods of interconnection: 1) Extend into foreign country 2) reciprically peer in each others coountry 3) Half-Circuit peering 4) Buy an ISP with Tier 1 peering in the foreign ecosystem The Peering Coordinators shared nine lessons, things they wish they had known before expanding into Asia: 1) Tier 1 ISPs will not want to peer in their home market 2) There are several challenges peering in Asia 2a) Peering Process is different - peering@ does not uniformly work 2b) Many language zones 2c) Many time zones - logisitics issues here 2d) Oceans add cost, latency - transpacific transport prices estimated in the paper 2e) local loops still a major expense 2f) transit prices highly variable across countries in Asia - transit street prices quoted 3) Creative peering deals 4) Watch for Inadvertent Tromboning 5) Local Presence is required 6) Separation of Int'l Transit from Domestic Transit in some markets 7) Content that transcends the language barrier is not allowed to be hosted in country 8) No true regional content in Asia 9) Content Peering works in Asia 10) Understand the Internet Peering Ecosystem prior to building in On this last point, the paper enumerates several data points including naming the Tier 1's, the emerging broadband players, the peering inclinations, etc. for Hong Kong, Tokyo, Sydney and Singapore. Some of the Yahoo!Broadband activities for example in Japan show that some places in Asia are leap-frogging the US in terms of next-gen access bandwidth. How would you like 40Mbps DSL access for $50/mo or FTTH for $100/mo? I'm at the point where I'd like to do a dozen or so final paper walk-throughs, where I email the white paper and walk interested folks through the paper to solicit comments and suggestions. I've found that talking people through the paper is this is the most effective way for me to make sure the paper (and subsequent presentation) flow well, and also increases the likelihood of me getting some feedback. If you have any interest in the topic and can spare about 30 minutes for a walkthrough over the next few weeks, send me a note and let's schedule a time I can talk you through the paper. I'm also planning a group walk through (a concall basically) towards the end of the month. As with all the Peering White Papers, this one will be made freely available to the community when it is finished. Bill Here are the other Peering White Papers that are available (google for "William B. Norton" to find them on the web, or email [EMAIL PROTECTED] with the white paper title in the subject line and I'll send you the latest version of the white paper.) 1. Interconnection Strategies for ISPs documents two dominant methods ISPs use to interconnect their networks. Over 200 ISPs helped create this white paper to identify when Internet Exchange Points make sense and the Direct Circuit interconnect method makes sense. Financial Models included in the paper quantify the tradeoffs between these two methods. All relevant data points are footnoted as to source. 2. Internet Service Providers and Peering answers the questions: "What is Peering and Transit? What are the motivations for Peering? What is the ISP Peering Coordinators Process for obtaining peering? What are criteria for IX selection?" 3. A Business Case for Peering builds upon the previous white papers but focuse
Re: New VOIP Peering/Interconnection Mailing List Announcement
is Paul is volunteering to host this (perhaps on peering.com)? At 06:49 PM 5/14/2004 +, Paul Vixie wrote: [EMAIL PROTECTED] (Daniel Golding) writes: > Open exchange of ideas is the goal! > > Please feel free to forward this announcement freely. > > Post message: > [EMAIL PROTECTED] > > Subscribe: > [EMAIL PROTECTED] could you put the list someplace else? i don't accept e-mail from yahoo here, since they don't do any kind of permission/verification and i got tired of JHD. which is too bad since i'm very interested in the topic of this mailing list. if you need a place to host a mailing list, i could ask around at my day job. -- Paul Vixie
Re: What percentage of the Internet Traffic is junk?
At 01:56 PM 5/5/2004, Marshall Eubanks wrote: Look at Table's 6, 7 and 8 - email, for example, is 1/2 %, so even if all email is spam, it's not that big a flow. Unidentified is typically about 30%, but most of that is probably file sharing. Thanks Marshall - a few others have said (paraphrasing): On average we have seen about 30% by packets (but only 10% by bandwidth) are junk, with higher %'s during major attacks and worm infestations. For those who say things like "can't define 'junk' precisely", I would agree, but I think we also can agree that we all have a general idea of what junk is. Just looking for round #'s really. It isn't 0%, and it isn't 90% (although it seems that way sometimes). I would also agree that it would be valuable for the community to track this # over time. You can't manage it if you can't measure it. Bill My opinion, from looking at these tables, is that probably little is junk, at least in the eye's of the receiver. Regards Marshall Eubanks On Wed, 05 May 2004 13:17:45 -0700 "William B. Norton" <[EMAIL PROTECTED]> wrote: > > At 12:55 PM 5/5/2004, Steve Gibbard wrote: > > >If a few of you can stop being so pedantic for a second, the definition > >looks pretty easy to me: traffic unlikely to be wanted by the recipient. > >Presumably, if it's being sent that means somebody wanted to send it, so > >the senders' desires are a pretty meaningless metric. > > Thanks Steve - good point. I have to believe that some of those that have > solutions to some of these problems have made *some* measures so they can > quantify the value of their solution. > > > >The harder pieces are going to be defining what traffic is unwanted in a > >way that scales to large-scale measurement. Worm traffic is presumably > >measurable with Netflow, as are various protocol-types used mainly in DOS > >attacks. Spam is harder to pinpoint by watching raw traffic, but perhaps > >comparing the total volume of TCP/25 traffic to the SpamAssassain hit > >rates at some representative sample of mail servers could provide some > >reasonable numbers there. > > Yea, we can't get absolute #'s, but I think it would be helpful to have a > defensible approximation. > > > >So, any of you security types have a list of the protocols that are more > >likely to be attack traffic than legitimate? > > Or maybe those in the Research Community that have been doing traffic > capture and analysis? > > > >-Steve > > > >On Wed, 5 May 2004, Mike Damm wrote: > > > > > > > > > > > Very very very near to, but not quite 100%. Since almost all of the traffic > > > on the Internet isn't sourced by or destined for me, I consider it junk. > > > > > > Also remember that to a packet kid, that insane flood of packets destined > > > for his target is the most important traffic in the world. And to a > > spammer, > > > the very mailings that are making him millions are more important than > > > pictures of someone's grandkids. > > > > > > I guess my point is junk is a very relative term. A study would need to > > > first be done to identify what junk actually is, then measuring it is > > > trivial. > > > > > > -Mike > > > > > > -Original Message- > > > From: William B. Norton [mailto:[EMAIL PROTECTED] > > > Sent: Wednesday, May 05, 2004 11:21 AM > > > To: [EMAIL PROTECTED] > > > Subject: What percentage of the Internet Traffic is junk? > > > > > > > > > With all the spam, infected e-mails, DOS attacks, ultimately blackholed > > > traffic, etc. I wonder if there has been a study that quantifies > > > > > > What percentage of the Internet traffic is junk? > > > > > > Bill > > > >
RE: What percentage of the Internet Traffic is junk?
At 12:55 PM 5/5/2004, Steve Gibbard wrote: If a few of you can stop being so pedantic for a second, the definition looks pretty easy to me: traffic unlikely to be wanted by the recipient. Presumably, if it's being sent that means somebody wanted to send it, so the senders' desires are a pretty meaningless metric. Thanks Steve - good point. I have to believe that some of those that have solutions to some of these problems have made *some* measures so they can quantify the value of their solution. The harder pieces are going to be defining what traffic is unwanted in a way that scales to large-scale measurement. Worm traffic is presumably measurable with Netflow, as are various protocol-types used mainly in DOS attacks. Spam is harder to pinpoint by watching raw traffic, but perhaps comparing the total volume of TCP/25 traffic to the SpamAssassain hit rates at some representative sample of mail servers could provide some reasonable numbers there. Yea, we can't get absolute #'s, but I think it would be helpful to have a defensible approximation. So, any of you security types have a list of the protocols that are more likely to be attack traffic than legitimate? Or maybe those in the Research Community that have been doing traffic capture and analysis? -Steve On Wed, 5 May 2004, Mike Damm wrote: > > > Very very very near to, but not quite 100%. Since almost all of the traffic > on the Internet isn't sourced by or destined for me, I consider it junk. > > Also remember that to a packet kid, that insane flood of packets destined > for his target is the most important traffic in the world. And to a spammer, > the very mailings that are making him millions are more important than > pictures of someone's grandkids. > > I guess my point is junk is a very relative term. A study would need to > first be done to identify what junk actually is, then measuring it is > trivial. > > -Mike > > -Original Message- > From: William B. Norton [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 05, 2004 11:21 AM > To: [EMAIL PROTECTED] > Subject: What percentage of the Internet Traffic is junk? > > > With all the spam, infected e-mails, DOS attacks, ultimately blackholed > traffic, etc. I wonder if there has been a study that quantifies > > What percentage of the Internet traffic is junk? > > Bill >
What percentage of the Internet Traffic is junk?
With all the spam, infected e-mails, DOS attacks, ultimately blackholed traffic, etc. I wonder if there has been a study that quantifies What percentage of the Internet traffic is junk? Bill
Peering BOF VII Meeting Minutes (NANOG 30 Miami)
Hi all -
For those of you who could not attend the BOF, here are my notes from the
Peering BOF. Comments welcome -
Peering BOF VII - NANOG 30 - Miami
2/10/2004 7PM
Moderator: William B. Norton
We were at capacity in the room and started right at 7PM.
The Great Debate
--
The Peering BOF started with a refereed debate on
"Restrictive Peering Policy: Makes Business Sense vs. Counter Productive."
The debate at the Peering BOF was a first miniature stab at a Internet
Operations debate focusing on an emotional charged topic, one area
perceived as being concealed by NDAs, hidden agendas, greedy corporate
interests, ill will: Restrictive Peering Policies. People attending the
BOF heard in about 20 minutes some pretty good arguments why such a policy
made business sense, and why it was also counter productive.
I opened sharing some of the controversy, sharing the definitions of
peering, transit, restrictive, selective and open peerign policies, etc.
These should be on-line at http://www.nanog.org/mtg-0402/norton.html
Vijay Gill (AOL) agreed to present the Affirmative Case, that Restrictive
Peering Policies make business sense. He argued that Peering Policies are
not emotional, not personal, but black & white decisions based on
economics, a calculation on a spreadsheet. Peering is not free and there
are scaling issues and associated operational costs that must be justified.
Peering is also somewhat of a threat to revenue production as well, since
peering between customers bypasses the ISP.
Avi Freedman (Akamai) agreed to present the Counter Case, that Restrictive
Peering Policies are counter-productive. He pointed to the overhead
associated with peering as being a rare exception; peering sessions tends
to stay up. Avi pointed to the counter productive aspects of a Restrictive
Peering Policy:
1) It inspires the wrong thing. Sprint will never peer with anyone that
ever bought transit from them, and that in fact drives the "End-run, peer
with Sprint's customers" behavior that lowers revenue for Sprint.
2) Many large networks lose revenue by not peering more openly, as it turns
off customers, and
3) Performance improvements from peering widely *increase* revenue. Avi
eluded to his AboveNet experience where he witnessed
a) customer behavior: customers spending more time on line because of a
better experience,
b) lower latency and lower packet loss lead to the TCP window opening up
faster means more data is exchanged, thus leading to more $ for those that
charge on a per-Mbps basis. Better performance drives more revenue.
Vijay countered with the "reducto ad adsurdium" argument; the case of
peering with each of 200 million laptops computers running zebra. The
overhead of peering with 200M laptops would certainly fail the cost benefit
analysis. As far as the performance argument, he claimed that the top
visited sites from AOL show little performance difference ("imperceptible")
between paths reached across peering versus transit links. Finally, he
dismissed the selection of an ISP based on Peering Policy, claiming it is
today all driven by price. The peering decision is business and
mathematical, black & white, a pure economic decision that does what is
best for the company.
Avi finished up by dismissing the peer-with-all-laptops as not reasonable,
not what anyone is advocating, not what is being debated. He argued that
the debate is really speaking to the reasonable middle ground case, where
both parties have infrastructure deployed. Around 50% of customer traffic
will go around you if you do not peer more openly. Restrictive Peering
Policies analysis must include the opportunity cost of lost business and
lost revenue, a more difficult calculation to make but one that ultimately
shows Restrictive Peering policies as counter productive.
VERDICT: The audience voted on "which side presented the more compelling
case". The winner: Restrictive Peering Policies are Counter Productive
(35-43). (Editor's note: this was much closer than I think most people
expected; the audience at the Peering BOFs are generally open or selective
peers, and have or expect to be stubbed if trying to peer with a
Restrictive Peering Policy peer. Both sides presented a good case.)
Cable & Wireless: A Tier 1 Peering Policy evolution & C&W migration to Savvis
--
After the debate, Peter Jansen (Peering Coordinator for C&W) volunteered to
share with the audience a bit about the evolution of the C&W Peering Policy
and what led to their current Peering Policy. He positioned a restrictive
policy as a natural outcome of commercial interests (peering costs money)
and but said that they will in fact
Peering BOF VII - Peering Personals is Full
Hi all - At this point the Peering Personals part of the Peering BOF is full - please do not send any more RSVPs. Since there was confusion over this point the last time, there is no need to RSVP to *attend* the Peering BOF, only to participate in the Peering Personals during the second half of the BOF. That is the part where Peering Coordinators introduce themselves with a screen behind them indicating where they peer, their AS#, e-mail address, etc. This is the part of the BOF where we can not (for time reasons) fit anymore speakers. I could use a couple people though to help manage the logistics (and security ;-)) for the opening "Great Debate" on "Restrictive Peering Policies; Makes Business Sense vs. Counter-productive" however. Send me an e-mail if you can help. Also, if you think there are some key questions that need to be answered by the debaters, or further comments or suggestions regarding the BOF, I'd love to hear them as well. See you all in Miami - Bill
Peering BOF: Announcing The Great Debate
Hi all - Restrictive Peering Policies: The Great Debate --- Monday Evening at the upcoming Peering BOF at NANOG 30 in Miami we are trying something new: at the beginning of the Peering BOF there will be "A Great Debate" on the topic of Restrictive Peering Policies. Taking the Pro: side is Vijay Gill who will argue "A Restrictive Peering Policy makes business sense." Taking the Con: side is Avi Freedman who will argue "Restrictive Peering Policies are counter productive." Note: The views presented do not necessarily represent the views of the individuals or their employers; the debaters have been coached to present the most compelling case possible given the following rough peering policy classifications and definitions: Open means that the entity will generally agree to peer with anyone in any single location with no prerequisites. Selective means that the entity will generally peer but there are some prerequisites (such as meeting in multiple Interconnect Regions, with a minimum traffic volume, not to exceed a certain In/Out traffic ratio, etc). The Peering Policy documents these prerequisites, which, once met, generally lead to peering. Restrictive means that the entity is generally not open to new peering. The Peering Policy documents extremely difficult to meet peering prerequisites, with the unstated purpose of denying peering. Peering is defined as a business relationship whereby two entities reciprocally and freely exchange access to each others customers. Transit is defined as a business relationship whereby one entity sells access to the *entire* Internet to another. There are of course variations of the above including Paid Peering (exchange of access to each others customers with some form of settlement) and Partial Transit (one entity sells access to part of the Internet, typically broader than just their customers). The format of the Debate: Coin Flip to decide who goes first Side A presents their position (3.5 minutes) Side B presents their position (3.5 minutes) Side A counters and/or reinforces their own position (3.5 minutes) Side B counters and/or reinforces their own position (3.5 minutes) Side A Summation (3.5 minutes) Side B Summation (3.5 minutes) The audience will vote "Who makes the more compelling case" to determine the winner of the debate. My hope is that this will focus the light on an issue that seems to have always caused a great deal more heat than light in the Peering Coordinator Community. Perhaps we will see that there are indeed reasonable and rational arguments on both sides of this debate. Following the debate will be Peering Personals, giving Peering Coordinators a chance to talk about their network, what they look for in a peer, why others should want to peer with them, etc. as per http://www.nanog.org/mtg-0402/norton.html This has proven to be an effective way of pulling together the community of Peering Coordinators, hopefully resulting in more peering sessions. I think you will find this debate and the Peering BOF highly entertaining and maybe even educational ;-) Cheers - Bill PS - If you are a Peering Coordinator and would like to take part in the Peering Personals, I have a handful of slots left. Please fill out the form at the URL above and send it to [EMAIL PROTECTED] Thanks!
Re: Evolution of the U.S. Peering Ecosystem v1.1
At 06:23 PM 12/5/2003 -0500, [EMAIL PROTECTED] wrote: > 1) The Cable companies are peering (with Tier 2s and each other) in a > *big* way That's probably why ATDN depeered ~20 networks over last few months, while Comcast and Charter do not peer at all. I had not heard that. As for Comcast and Charter, I would add the word "yet." > 2) The Large Network Savvy Content Companies are getting into peering in > a *big* way With transit bandwidth at 20k$/GE, and Equinix shared fabric now priced at nearly half that, I don't see that many "content companies" peering all that much. The phrase, "You get what you pay for" comes to mind. There is real difference between transit services IMHO. Also, you rarely use the full gigE in these types of arrangements, so you need to be a little careful doing the "simple math." Having said all that, the transit price pressures are certainly there and it does make the peering financial argument a little tougher. I've heard the argument that "Peering is better than transit. It ought to cost more." > 3) The Large Network Savvy Content Companies are getting their content > directly onto the Cable companies eyeball networks by peering > relationships. I wish. Well then you shall receive. The large guys do peer (Yahoo!, MSN, Google, EA, Sony Online, etc.) in a very big way. I expect too these guys are simply leading the charge - those content companies large enough to employ a network engineering staff that can do the math will be following as well. It is not only a financial motivation; peering provides greater control over routing and is often seen as a performance enhancing strategy. Folks like Yahoo! seem to emphasize the end-user performance improvements over the financial savings these days. Out of big eyeball networks, only SBC has reasonably open policy, rest are attempting to force "content networks" into paid peering arrangements using restrictive ratio requirements Hmm. SBC has what I call a "Selective Peering" inclination; they will peer with you if you meet certain criteria. The cable companies are all different of course but generally seem to have migrated from a "No Peering" inclination (when @Home ran things for them the cable companies didn't peer) to an "Open Peering" inclination to reduce costs (and to deal with Kazaa traffic), to a "Selective Peering" inclination. I see this last step as an operations sanity motivation; peer with those who have at least 10M of traffic to exchange on a couple coasts. If you don't have that much traffic it may not be worth the time to configure the session, and when things break it may be more hassle than it is worth. The ones who are a bit different are the "Restrictive Peering Inclination" folks; those who have a peering policy articulated solely so they can say "No" with a reason. Rather than deal with the hassle of peering requests, some of these guys don't articulate their Peering Inclination in the form of a Peering Policy. "We have all the peering that we need" is the attitude, and, not to open a can of worms here, but it is a business rational attitude. Bill -- Alex Pilosov| DSL, Colocation, Hosting Services President | [EMAIL PROTECTED](800) 710-7031 Pilosoft, Inc. | http://www.pilosoft.com
Peering BOF VII at NANOG 30 in Miami
Hi again -
We have approval to run the Peering BOF again at the upcoming NANOG 30 in
Miami ! Appropriate attire (Hawaiian Shirts) is required ;-)
So, now my job is to draft Peering Coordinators to stand up, introduce
themselves, say a few words about their peering requirements, why you
should peer with them, etc. in order to facilitate peering. While you
introduce yourself I will have a screen behind you that has the answers to
the questions below. At the end of the BOF we break for beers and, if
history is any indication, peering coordinators meet each other, exchange
cards and get some direct peering going.
BTW - This time we may add an agenda topic, perhaps a brief debate
"Tier 1 Peering Policies: Defensible?".
Taking the Affirmative will be ___, taking the Negative will be __.
(Both TBD, debaters being secretly solicited)
Bill
--- For Peering Coordinators participating in Peering Personals
---
The Peering BOF focuses on meeting other Peering Coordinators using a
methodology we call "Peering Personals." We solicit Peering Coordinators
(before the meeting), asking them to characterize their networks and
peering policies in general ways ("content heavy" or "access (eyeball)
-heavy," "Multiple Points Required" or "Will Peer anywhere," "Peering with
Content OK," etc.). From the answers we will select a set of ISP Peering
Coordinators to present a 2-3 minute description of their network, what
they look for in a peer, etc., allowing the audience to put a face with the
name of the ISP.
At the end of the Peering BOF, Peering Coordinators will have time to speak
with Peering Coordinators of ISPs they seek to interconnect with. The
expectation is that these interactions will lead to the Peering
Negotiations stage, the first step towards a more fully meshed and
therefore resilient Internet.
If you are a Peering Coordinator and wish to participate in this BOF,
please fill out the following form and e-mail it to
[EMAIL PROTECTED] with
Subject: Peering BOF VII
We will only be able to accommodate maybe 20 peering coordinators this time
so please RSVP early!
- clip here
--
Name:
Title:
Company:
AS#:
Check each that applies:
___ We are an ISP (sell access to the Internet)
-- OR --
___ We are a Non-ISP (content company, etc.)
___ We are generally more Content-Heavy
-- OR --
___ We are generally more Access-Heavy
___ Peering with Content Players or Content Heavy ISPs is OK by us
___ We generally require peering in multiple locations
___ We will peer with anyone in any single location
___ We have huge volumes of traffic (lots of users and/or lots of content)
(huge: > 1 Gbps total outbound traffic to peers and transit providers)
___ We have a global network
___ We require Contracts for Peering
List all Current Peering Locations:
1)
2)
3)
:
List all Planned (3-6 mos) Peering Locations:
1)
2)
3)
:
Privacy Notice: This information is made available only to me, William B.
Norton, as an individual, and will be used only for facilitating the BOF
and making up the screen behind the speakers. People in the audience are
certainly welcome to jot down information presented at the BOF, but the
information will not be made available in any other form to anyone.
Hopefully this clarifies things and will avoid the controversy of the past.
To give you an idea of previous Peering BOFs please see
---
Peering BOF VI, NANOG 27, Phoenix, February 2003.
http://www.nanog.org/mtg-0302/norton.html
Peering BOF V. NANOG 25, Toronto, June 2002.
http://www.nanog.org/mtg-0206/norton.html
Peering BOF IV. NANOG 23, Oakland, October 2001.
http://www.nanog.org/mtg-0110/norton.html
Peering BOF III. NANOG 20, Washington, DC, October 2000.
http://www.nanog.org/mtg-0010/peering.html
Peering BOF II. . NANOG 19, Albuquerque, June 2000
http://www.nanog.org/mtg-0006/peering.html
Peering BOF I. NANOG 17, Montreal, Oct 1999
http://www.nanog.org/mtg-9910/peering.html
Evolution of the U.S. Peering Ecosystem v1.1
Hi all - Thanks to those who provided comments to the last white paper draft of "The Evolution of the U.S. Peering Ecosystem". I've made most of the changes and added the data points as suggested, so I am now ready to send out the document more broadly. Lots of acknowledgements in the acknowledgements section now - Thanks!! In a nutshell, the Evolution of the U.S. Peering Ecosystem introduces the notion of the Internet as a set of Regional Peering Ecosystems, each with its own - Tier 1s (who have access to the entire regional routing table solely through peering relationships), - Tier 2s (broadly all other ISPs), and - Content Providers These players are modelled with characteristics (upstream transit links, peering links, etc.) and their motivations (described as Peering Inclinations (Open, Selective, Restrictive, or No Peering) articulated in Peering Policies) that can predict roughly their behavior in the Peering Ecosystem. The "Evolution" of the U.S. Peering Ecosystem is the result of five forces: 1) The so-called dot.bomb - economic collapse of the telecom sector 2) The emergence of a large scale used equipment market 3) The exponential growth of Kazaa traffic costing eyeball networks 4) The failure of @Home - cable companies provide Internet services themselves 5) The rapid decline in transport and transit prices The three major changes roughly are: 1) The Cable companies are peering (with Tier 2s and each other) in a *big* way 2) The Large Network Savvy Content Companies are getting into peering in a *big* way 3) The Large Network Savvy Content Companies are getting their content directly onto the Cable companies eyeball networks by peering relationships. These are significant changes due to the volume of traffic exchanged, the amount of money being saved by avoiding intermediary transit providers, and the performance implications of these direct connections. As promised, if you are interested in this stuff I will gladly send you a copy of the latest draft, v1.1. I'm working on the same exploration for the Asia Pacific Peering Ecosystems (hence the APRICOT Peering Track note I sent out earlier this week). Hope this helps - Bill
APRICOT 2004 : Speakers for Peering and Internet Exchange Track
Hi all - You may have heard that there will be a new one-day track on Peering and Internet Exchanges at the upcoming APRICOT in Kuala Lumpur in February. I'd like to describe the track in brief and solicit a few more Peering Coordinators / Network Architects / Network Engineers as speakers for the track. (See http://www.apricot2004.net/ for information on the broader APRICOT 2004 conference). This Peering and Internet Exchange Track is intended to facilitate regional ISP Peering by providing a forum for Peering Coordinators to meet each other and share information about Asia Pacific peering. We will facilitate this with several panels, each one focused on peering in a particular country. If you are a Peering Coordinator with experience peering in the AP Region, please send me an e-mail answering the questions below. I will use this information to match panelists together into panels focused on certain AP countries. Thanks! Bill --- Speaker Questionnaire If you are able and willing to speak at the Feb 2004 APRICOT Peering and Internet Exchange Track, please fill out the following form and e-mail to William B. Norton at [EMAIL PROTECTED]: Name: __ Title: ___ Company: ___ AS # _ In what country do you live? _ Email Address: __ We will select a set of panelists based on the answers to the questions below: 1) In what countries do you peer? 2) In which AP Country do you have peering experiences that you can share with the audience? 3) Answer as many of the questions below as applicable, as you would present as part of your presentation a) We are looking for speakers that have found interesting or unique country peering ecosystem characteristics. Have you uncovered any unexpected or interesting/unique peering ecosystem characteristics in any of the countries where you peer? Please give a few examples. b) We are also looking for unusual or unexpected traffic patterns. For example, a large amount of Japan traffic is ultimately destined to and from Brazil! Identifying and explaining this type of inter-country traffic would be a good set of data to launch a presentation. Do you have any of this type of data that you can share? c) What are the problems and challenges that you face as you build peering infrastructure into various AP countries? d) If other Peering Coordinators follow your footsteps into regions in which you now peer, what would you tell them: · is information that you would have liked to have had? · Are unexpected problems that you faced? e) What are the emerging trends that you see in the peering ecosystems in which you operate (transit and transport prices going up or down and the implications as you see them). /* William B. Norton <[EMAIL PROTECTED]> 650.315.8635 Co-Founder and Chief Technical LiaisonEquinix, Inc. */
The Evolution of the U.S. Peering Ecosystem
Hi all - I've been working on documenting some of the significant disruption from and aftermath of the Telecom collapse of 1999/2000, focusing specifically on the operations community and the Peering Ecosystem in particular. I spent a lot of time speaking with Peering Coordinators to document the first order effects and some of the second order effects of the bankruptcies. I found some pretty interesting and fundamental changes in how the Internet is interconnected. Several new players have had a huge impact on what I call the "Internet Regional Peering Ecosystem." I presented a draft of this research at the GPF VII in Ashburn, Virginia last month and would love to have a few more reviewers give it a read and provide feedback. I pasted the abstract below. Thanks! Bill Abstract A new Internet Peering Ecosystem is rising from the Ashes of the 1999/2000 U.S. Telecommunications Sector crash. Global Internet Transit Providers have gone bust and a critical broadband infrastructure provider has failed, leaving in their wake a large set of Internet players to fend for themselves to provide their customers with Internet services. A broad set of Service Providers that were once focused only on growing their market share (at any cost) now are bending down to shave pennies off of their cost structure. Those who can not prove the viability of their business model while satisfying their customer demands are out of business. In this paper we share research carried out over the last four years with hundreds of Peering Coordinators to document the recent chaotic evolution of the Peering Ecosystem. We do this by first defining the notion of an Internet Peering Ecosystem, an Internet Region and Interconnection Region. We find in each Internet Peering Ecosystem three distinct categories set of participants, each with their own sets of characteristics and corresponding motivations and interconnection dynamics. We describe four classes of Peering Inclinations as articulated in Peering Policies. The bulk of the paper however focuses on the Evolution of the U.S. Peering Ecosystem. Several key players, some abandoned by their service providers, have entered into the Peering Ecosystem and caused a significant disruption to the Ecosystem. Peer-to-Peer application traffic has grown to represent a significant portion of their expense. We describe five major events and three emerging dynamics in the Peering Ecosystem that have had and continue to have a disintermediation effect on the Tier 1 ISPs. In the appendix we share a simple mathematical Internet Peering Model that can be used to demonstrate this Peering Ecosystem evolution. While not complete or by any means precise, it does allow us to demonstrate the affect of these disruptions in the Peering Ecosystem. /* William B. Norton <[EMAIL PROTECTED]> 650.315.8635 Co-Founder and Chief Technical LiaisonEquinix, Inc. */
Re: (possible Flame bait) Backbone Building vs Transit purchasing
At 06:27 PM 3/21/2003 -0500, [EMAIL PROTECTED] wrote: > > *IS* there a common sense number or an equation (better) anyone has worked > > out to figure whether building a backbone (national/international) to > > peering points (i.e. extending an existing, operational service network) to > > improve/add peering vs continuing to buy transit? Take a look at the financial models in: http://arneill-py.sacramento.ca.us/ipv6mh/ABusinessCaseforISPPeering1.2.pdf > > If you are assuming that this is not about performance then surely this is a > very simple thing to work out? > > Cost of transit T = cost of transit/committed Mbs > Cost of peering P = (cost of: circuits+routers+colo+nap)/Mbs of actual traffic > Right - the comparison can get a little tricky because 1) Transit is typically charged on the 95th percentile measure of Megabits per second, vs. peering, which as you will see, is a monthly recurring, not a Mbps. 2) Transit fees vary depending on the commit rate (50Mbps, 100Mbps, etc.) and terms of the agreement (6mo, 1yr, 2yr, etc.) while peering costs are 1) Monthly Recurring circuit, colo, port and cross connect costs , and 2) Capital (Router) costs are typically allocated across a number of periods based on financial standards In "The Business Case for Peering" I ignored capital (router costs) as they were highly variable across the different ISPs I spoke with. Everyone had a different kit. In the "Do ATM-based Internet Exchange Points Make Sense Anymore?" white paper I spoke with more Peering Coordinators, made some assumptions based on their suggestions, and added the capital costs into the equation: http://www.deliandmarshall.com/docs/ATM-based.pdf The good news is that transport prices have dropped like a rock, with cut throat competition in the metro markets. Transit has also dropped substantially, while IX fees seem to have remained about flat. You have to get your own #'s and do the math, but the above papers can give you a good start, and the transport/ transit/IX fee #'s in this last paper are pretty recent (October 2002). > If P>T go and push your network out to the peering point it will save you money. > > Now.. at present your problem is that T is very low, and certainly lower than P > unless you are moving quite a lot of traffic.. 1Gb is a lot of traffic, so all > you need to do is to figure out the costs in getting to a NAP and how much > traffic you can shift. [skip] You are forgetting: salaries depreciation leases IRU financing expenses As for salaries, I spoke with the Peering Coordinator community about this and there was some debate about this, with some claiming that once the peering is set up, there is very little to do, so not much staff time was required to care and feed for the session. The claim was that the NOC was capable of servicing the needs of the peering session and 2nd level support was already in place if they couldn't. So the salaries were already covered in the network engineering budget. Others claimed that Peering is a local routing optimization, and in the worst case, it fails, they turn it off, and they go back to buying transit to reach those routes. ... etc etc etc Alex
Last Call for Participation: Peering BOF VI at NANOG
Several folks have asked me for a list of who is participating at the
Peering BOF VI at NANOG. Here is a list of who I have so far:
Steve Schecter Net Access Corp AS8001
Chris Malayter TDS Telecom AS4181
Celeste AndersonUSC/ISI AS226
Daniel Golding AOL/Time Warner AS1668
Shawn Solomon Indiana Telecom Net AS1767
Chris CaputoAltopia AS6456
Ingrid Erkman ICG AS2551
Matthew PloesselToneLinkAS26322
Ren Nowlin SBC AS7132, 5673, 5676, ...
Patrick Gilmore Akamai AS1
Louie Lee Equinix AS14609, 9989, 17819
Allison Feese BroadWing AS6395
Joe Provo RCN AS6079
Joe Klein AdelphiaAS19548
Pete KruckenbergUtah Educ Net AS210
We can take maybe 10 or so more Peering Coordinators. If you would like to
participate, please fill out the form and send it to [EMAIL PROTECTED]
http://www.nanog.org/mtg-0302/norton.html
For the BOF we'll have each Peering Coordinator stand up, introduce
themselves, talk about their network, what they look for in a peer and why
other ISPs/CPs should want to peer with them. On the screen behind them
will be their contact info (Name,email,AS/Company Peering URL) along with
answers to the questions about Peering Inclination (Content Heavy, Access
Heavy, Open Peering vs. Multi-Site requirements, etc.) and Current/Planned
Peering locations. This has proven to be a valuable way for Peering
Coordinators to meet each other and make that initial contact for
establishing peering. ( It is much easier when people recognize you from
your 3-5 minutes of fame.)
Comments/Suggestions welcome.
Bill
Date: Fri, 10 Jan 2003 08:16:02 -0800
To: [EMAIL PROTECTED]
From: "William B. Norton" <[EMAIL PROTECTED]>
Subject: Peering BOF VI at NANOG
Hi all -
If you are not a Peering Coordinator attending NANOG 27 then you needn't
read any further.
The 6th Peering BOF at NANOG will be held Monday night and focuses on
helping Peering Coordinators make contact with other Peering Coordinators
using "Peering Personals." We solicit Peering Coordinators (via this
e-mail), asking them to characterize their networks and peering policies
in general ways ("content heavy" or "access (eyeball) -heavy," "Multiple
Points Required" or "Will Peer anywhere," "Peering with Content OK,"
etc.). From the answers we will select a set of ISP Peering Coordinators
to present a 2-3 minute description of their network, what they look for
in a peer, etc., allowing the audience to put a face with the name of the
ISP. At the end of the Peering BOF, Peering Coordinators will have time to
speak with Peering Coordinators of ISPs they seek to interconnect with.
The expectation is that these interactions will lead to the Peering
Negotiations stage, the first step towards a more fully meshed and
therefore resilient Internet.
If you are a Peering Coordinator and wish to participate in this BOF,
please fill out the following form and e-mail it to [EMAIL PROTECTED] with
Subject: Peering BOF VI .
Name:
Title:
Company:
AS#:
Check each that applies:
___ We are an ISP (sell access to the Internet)
-- OR --
___ We are a Non-ISP (content company, etc.)
___ We are Content-Heavy
-- OR --
___ We are Access-Heavy
___ Peering with Content Players or Content Heavy ISPs is OK by us
___ We generally require peering in multiple locations
___ We will peer with anyone in any single location
___ We have huge volumes of traffic (lots of users and/or lots of content)
(huge: > 1 Gbps total outbound traffic to peers and transit
providers)
___ We have a global network
___ We require Contracts for Peering
Current Peering Locations: ___
Planned (3-6 mos) Peering Locations: ___
See you in Phoenix!
Bill
PS - This form is also on the NANOG web page at:
http://www.nanog.org/mtg-0302/norton.html
---------------
William B. Norton <[EMAIL PROTECTED]> 650.315.8635
Co-Founder and Chief Technical Liaison Equinix, Inc.
Re: US-Asia Peering
At 09:33 AM 1/10/2003 -0800, Bill Woodcock wrote: On Fri, 10 Jan 2003, Stephen J. Wilcox wrote: > In response to Randy and Bill(s), this seems to come down to a trade off of > commercial vs technical. A lot of us agree this is technically not the best way > and produces instabilities with the potential to take out major chunks of > internet but it is cheap and this means people will adopt this way of doing it, > unfortunately as this has now happened it means those opposed to the idea will > have to also consider this as an option if they are to compete. I don't think it's fair to characterize it as a trend... I mean, ten years ago, we were all (generalizing here) stupid enough to try these tricks. Fortunately, smarter people have come along since, and learned from our mistakes. There are also _vastly_ more people involved in the industry now than then, so it comes as no surprise that there are still some newbies trying this, despite all the lessons of the past. The good news is that although they're a quantitatively growing group, they're a shrinking _fraction_ of the whole. So that's evidence of some small progress in the state of knowledge. Fight the law of conservation of clue! -Bill Bill - the argument seems like Proof by Rigorous Assertion: I know it is a bad idea. I really really believe it is a bad idea. My friends say it's a bad idea. Not one that I know says it is a good idea. Therefore, and I can't emphasize this enough, in conclusion, it is a bad idea. If what you are saying is true, I'd really like to hear just a couple of insurmountable technical problems with WAN L2.5 infrastructure interconnecting IX switches. For the sake of argument and to clarify the discussion (Paul) let's make a few assumptions: 1) We are talking about an operations model where IX switches are operated by a single company. 2) The IX switches are interconnected by MPLS by a transport provider offering that service. 3) An ISP on one switch creates a VLAN for peering with ISPs on any of the other switches. This ISP VLAN is only for peering with the ISP that created this VLAN. Since he is paying for the VLAN traffic he has this right. 4) The cost of transporting the traffic between the switches is bourne by a transport provider who in turn charges the ISP that created the VLAN in question. I can articulate a half dozen reasons why this is a good idea. Please share with us why this is a such a bad idea. If it has been tried before, it would be helpful to point to specific the case and why it failed, the technical failure scenario. I'd like to hear why/how it was worse by the distance between switches. Bill
Peering BOF VI at NANOG
Hi all -
If you are not a Peering Coordinator attending NANOG 27 then you needn't
read any further.
The 6th Peering BOF at NANOG will be held Monday night and focuses on
helping Peering Coordinators make contact with other Peering Coordinators
using "Peering Personals." We solicit Peering Coordinators (via this
e-mail), asking them to characterize their networks and peering policies in
general ways ("content heavy" or "access (eyeball) -heavy," "Multiple
Points Required" or "Will Peer anywhere," "Peering with Content OK," etc.).
From the answers we will select a set of ISP Peering Coordinators to
present a 2-3 minute description of their network, what they look for in a
peer, etc., allowing the audience to put a face with the name of the ISP.
At the end of the Peering BOF, Peering Coordinators will have time to speak
with Peering Coordinators of ISPs they seek to interconnect with. The
expectation is that these interactions will lead to the Peering
Negotiations stage, the first step towards a more fully meshed and
therefore resilient Internet.
If you are a Peering Coordinator and wish to participate in this BOF,
please fill out the following form and e-mail it to [EMAIL PROTECTED] with
Subject: Peering BOF VI .
Name:
Title:
Company:
AS#:
Check each that applies:
___ We are an ISP (sell access to the Internet)
-- OR --
___ We are a Non-ISP (content company, etc.)
___ We are Content-Heavy
-- OR --
___ We are Access-Heavy
___ Peering with Content Players or Content Heavy ISPs is OK by us
___ We generally require peering in multiple locations
___ We will peer with anyone in any single location
___ We have huge volumes of traffic (lots of users and/or lots of content)
(huge: > 1 Gbps total outbound traffic to peers and transit providers)
___ We have a global network
___ We require Contracts for Peering
Current Peering Locations: ___
Planned (3-6 mos) Peering Locations: ___
See you in Phoenix!
Bill
PS - This form is also on the NANOG web page at:
http://www.nanog.org/mtg-0302/norton.html
-----------
William B. Norton <[EMAIL PROTECTED]> 650.315.8635
Co-Founder and Chief Technical Liaison Equinix, Inc.
Re: US-Asia Peering
At 08:14 PM 1/9/2003 -0800, Randy Bush wrote: > Well, first I think we need to agree that there are two different cases here: > 1) interconnecting IXes operated by the same party, vs. > 2) interconnecting IXes operated by different parties. > > In the first case an IX operator can shoot himself in the foot, but there > is only one gun and one person, so you can easily figure out why the foot > hurts. well, now we know you have ever had to debug a large L2 disaster Randy - You snipped out what I said out of context. Below is the complete paragraph (and admittedly I should have said "relatively easily" rather than "easily".) The point is that I don't think we are talking about interconnecting switches operated by different parties, and I think you would agree that if it is difficult diagnosing problems with a single large scale l2 fabric, it is even more difficult with multiple administrative domains. That was the point. Original Paragraph: >In the first case an IX operator can shoot himself in the foot, but there is only >one gun and one person, so you can easily figure out why the foot hurts. >In the latter case, there are more people with more guns. Without perfect >information distributed among the operators, this is clearly a more dangerous >situation and diagnosing/repairing is more difficult and time intensive. I believe >we are really talking about the first case. Woody - I'd still like to hear about the failures "in every prior instance". >> clearly, interconnecting their exchange points to create a richly- >> connected Internet 'core' is a natural progression if their >> customers don't complain too loudly. >> not that it's a bad long-term plan... >Actually, it is. It's failed in every prior instance. Thanks.
Re: US-Asia Peering
At 06:07 PM 1/9/2003 -0800, Randy Bush wrote: > Where the same pseudo wire provider connects to say LINX, AMSIX, > DECIX your only a little way off having an interconnection of > multiple IXs, its possible this will occur by accident .. and l2 networks scale s well, and are so well known for being reliable. is no one worried about storms, spanning tree bugs, ... in a large multi-l2-exchange environment? this is not a prudent direction. Well, first I think we need to agree that there are two different cases here: 1) interconnecting IXes operated by the same party, vs. 2) interconnecting IXes operated by different parties. In the first case an IX operator can shoot himself in the foot, but there is only one gun and one person, so you can easily figure out why the foot hurts. In the latter case, there are more people with more guns. Without perfect information distributed among the operators, this is clearly a more dangerous situation and diagnosing/repairing is more difficult and time intensive. I believe we are really talking about the first case. Secondly, some of the issues of scaling l2 infrastructure have been addressed by VLANs, allowing the separation of traffic into groups of VLAN participants. This reduces the scope of an L2 problem to the VLAN in use. Since the role of the IX operator is to provide a safe stable scaleable etc. interconnection environment, distributed VLANs are a tool that can help extend the peering population while mitigating the risk of any single ISP from wrecking the peering infrastructure. Bill randy
Re: US-Asia Peering
At 10:35 AM 1/3/2003 -0800, Bill Woodcock wrote: > clearly, interconnecting their exchange points to create a richly- > connected Internet 'core' is a natural progression if their > customers don't complain too loudly. > not that it's a bad long-term plan... Actually, it is. It's failed in every prior instance. I'd like to understand your viewpoint Bill. The LINX consists of a handful of distributed and interconnected switches such that customers are able to choose which site they want for colo. Likewise for the AMS-IX and a handful of other dominant European exchanges. By most accounts these are successful IXes, with a large and growing population of ISPs benefiting from the large and growing population. So I don't see the failure cases. It's one of the many, many ways in which exchange points commit suicide. I'd love to see a list of the ways IXes commit suicide. Can you rattle off a few? -Bill
US-Asia Peering Research Request
Along a same lines... I'm working on the next white paper on "Peering in Asia" and I'd like to again ask for data points from the community. To start... I remember back at APRICOT in 1999 that some folks (Dave Rand and colleagues maybe?) were talking about an initiative to provide an AP Peering Ring... To help with this research (as before I'll be glad to share the results with anyone who is interested) do any of you have URLs/Research Papers that show: (1) How peering is done in the AP region today? I've heard generally of a) the AP Mesh of 1/2 circuits between countries PTTs, and b) the AP Country Peering Point(s) but I'm curious if anyone has documented down deeper than that generalization. (2) Past Initiatives for AP Region Peering (or AP-US Peering) that have succeeded or (better yet) have failed? (3) Rough #s for Transit Fees in the AP Region. Several of you shared price points of $300-$400USD/Mbps for ~10-50Mbps, maybe $100 for 1+Gbps in Japan. If anyone has done a survey for the AP Region in general as I did in the past in the U.S. markets I'd love to see it. The U.S. curve for example seems to be a stepped function, starting at about $300/Mbps for a few Mbps down to sub $100/Mbps when you get towards the Gbps range. I'd like to hear what AP ISPs are seeing. Thanks in advance and I'll write this up and let you know what I find out. As before, I'll credit the sources that provide the data points and honor anonymity as requested. Hopefully the end result with be of value to the community. Contact information below. Bill ----------- William B. Norton <[EMAIL PROTECTED]> 650.315.8635 Co-Founder and Chief Technical Liaison Equinix, Inc. Yahoo Instant Messenger ID: WilliamBNorton
Re: US-Asia Peering
Thanks all for your responses (both public and private). Several folks wanted to know what I found out so... I heard from a couple companies that are operating wide area distributed peering architectures today. They claim that the biggest issues has been the perception among prospects that "ethernet isn't supposed to do that (extreme long distance)." I'd love to hear more experiences both pro/con. (I have to admit I was surprised that *transoceanic ethernet* as a shared peering transport did not have serious issues. I would have expected that the time delay from the time a broadcast was transmitted to the time it was heard would have been an issue somehow, or some such interesting problems would come up.) Several VLAN configuration issues came up as a design consideration for wide area peering infrastructure. For example a) a VLAN for each peering session vs. b) one VLAN per each customer to which others "subscribe" and peer across vs. c) a global VLAN which nobody likes. There are policy and design tradeoffs with each of these that touch on the limitation of 4096 VLANs . As for transport, MPLS framing of ethernet seems to work well. The question of tunneling transport over existing transit connections has proven effective to trialing but may be more expensive as the traffic volume increases. Running circuits of dedicated access can reduce the risk of running out of capacity on a "shared" transit or MPLS IX interconnect fabric. As for the operator of the transport between distributed switches, Joe Provo is correct that it need not be the IX operator. IX neutrality generally means that the IX Operator is not aligned with any one participant in the IX, but rather is working to the benefit of all of it IX participants. If an IX Operator's actions unnecessarily favor or harm one participant over another, then neutrality may an issue. Extending the population of an IX by using a distributed architecture doesn't necessarily clash with this neutrality principle, especially if doing so is solely for extending peer-peer interconnection. And no, this is not a new idea; the LINX, AMSIX, etc. have been doing this for a long time and the key seems to be that the IX switches are under one autonomous control. Bill
US-Asia Peering
Hi all - I understand that there is a real glut of AP transoceanic capacity, particularly on the Japan-US cable where twice as much capacity is idle as is in use. This has sent the price point down to historic levels, O($28K/mo for STM-1) or less than $200/Mbps for transport! This is approaching an attractive price point for long distance peering so, just for grins,... Are there transport providers that can provide a price point around $100/Mbps for transport capacity from Tokyo to the U.S. (LAX/SJO) ? What are the technical issue with extreme long distance (transoceanic) peering? In particular, what are the issues interconnecting layer 2 switches across the ocean for the purposes of providing a global peering cloud using: 0) vanilla circuit transport to interconnect the switches 1) MPLS framed ethernet service to interconnect the switches 2) tunnelled transport over transit to interconnect the switches Thanks in advance. Bill
Re: Fwd: Alternative to NetFlow for Measuring Traffic flows
At 04:36 AM 12/18/2002 +, Sean M.Doran wrote: I have found peering to have additive value; a lot of 1-2 Mbps peering sessions can save as much money for you as a single large traffic peer. The more traffic, the stronger the case for peering. Sadly, this completely ignores the cost of implementing and maintaining peerings. BGP does not exactly configure itself, and current routing technology is somewhat frail -- if it breaks, somebody has to pick up the pieces; if suboptimal routing results from a peering, somebody will have to go and tweak things. Since the Internet is not exactly static, these things creep up from time to time even after a peering is established. Agreed there is an incremental operational cost of peering. I've heard differing views on the importance of peering. One view is that Peering is a "Local Routing Optimization" for which an ISP always has the fallback of transit. Others view the peering session as very important to both peers for capacity, cost and performance reasons. These two different views tend to lead to different levels of resources and overhead. Most peers minimally expect (or require in BLPAs) each other to have a 24/7 NOC that works diligently to fix a broken peering session. Generally, this is a (financially) small incremental expense since the NOC staff and facilities etc. are already in place. An exhaustive itemization of the costs of any given peering is a of vital component of a cost-benefit analysis, particularly where much of the benefit is a reduction in monthly usage based billing costs, or a deferral of an upgrade of a flat rate contract, rather than installing new parallel connectivity to meet the demands of traffic growth. While such a list will vary from organization to organization, and some organizations may be tricky to complete, one can consider the primitive case of a transition from being singly homed to a transit provider to bring up an initial peering. Among the things to be considered: PA address space, BGP (and making sure that the routers can handle the load, and so can the people operating them), avoidance of accidental transit, what happens to the peering traffic when the peering fails from time to time (or fills up with growth traffic over time), NOC-to-NOC coordination, impact on actual and potential SLA offerings, and so on. The immediately subsequent peering may not make much of an impact on many of these however there are real incremental costs. Some of these may not rise linearly in proportion to the number of peers (e.g. there may be a step change), but rather may be a function of that number, the number of your routers which talk eBGP, and your overall traffic load. Here again I agree, all things considered equal, fewer sessions cost less to build and manage than more sessions. Thankfully, peering sessions generally tend to stay up, providing the benefits of peering 99.9..% of the time. I haven't heard many Tier-2 ISP Peering Coordinators (yet) complain that the ongoing operational overhead of their peering sessions was overly burdensome compared to the benefit they derive from their peering sessions. I have heard a few of the Tier-1 ISPs make these complaints however... but this may lead to a different conversation. I know a handful of cases where a broad peering strategy had a fairly clear negative impact on the bottom line even though the saving in transit fees was both directly measurable and large. Anecdotes are not general proofs, but do underline the uncertainty in your statement: "a lot of ... peering sessions CAN save as much money ... as a single large traffic peer". [emphasis mine] Unfortunately, discussions here seem to focus mostly on measuring the reduction in transit fees. Wouldn't it be nice if this could be coupled with reasonable discussion about the increase in other costs, and how for some networks these costs are much higher than for others? Tough to measure and quantify these incremental costs whereas the improvement in performance and cost savings are relatively easily measured and calculated. I seem to remember reading a paper about that... Sean.
Re: Alternative to NetFlow for Measuring Traffic flows
Quantifiable Proof and "Peering Profiles"...see below. At 08:53 PM 12/16/2002 -0800, Joe Wood wrote: On Mon, 16 Dec 2002, Joe Abley wrote: > I think the idea was to say "well, from the mrtg graph, the difference > between this circuit with all my _9327_ traffic and this circuit > without any _9327_ traffic, at what I might reasonably estimate their > peak time to be, looks to be about 2 megs or so". > > It's a pretty crude measure, but it does have the advantage of > requiring no more than mrtg and a route-map to set up. Right, it is crude, but in an economy where business decisions require "Quantifiable *Proof*", this is quantifiable and easy to do. Some Peering Coordinators are putting together business plans now for peering at the IX that includes the #'s of Mbps of peering traffic, and e-mail confirmation from the peers at the IX that they will indeed peer with them at the IX. Smart customers; if they exceed the breakeven point then peering makes sense. A lot more work up front than it used to be. It is also useful as a supplement to netflow statistics, as sort of a verification to your flow data. Sometimes due to design, operating conditions, etc netflow data is not always the most reliable and/or meaningful. As an example: You run two main types of border router platforms. On one platform you must sample netflow @ 1% due to performance limitations. On the other platform there is no sampling functionality built into the software. This creates an immediate skew of data, unless software is created to sample the flows coming off the second platform. Now take into account that your traffic is mainly outbound from your network, which means that you need to ignore vendor best practice and enable flow caching on your core (internal) facing interfaces to measure the traffic flowing out of your network. So, in order for you to get any kind of traffic statistics for a peer, you've got to spend many hours distilling data manually, doing AS aggregations, and create a possibly unstable networking environment. No big deal, right? It may be crude, but sometimes it can be the most reliable _available_ method to tell how much traffic is going to the ISP and ISP customers. Joe is absolutely right here, and this still represents a common scenario and problem for the peering community. Another approach I have been thinking about is to generate "Peering Profiles" for the community...here is how it works. Let's say I work with a few Internet Gaming companies and find that the netflow stats show a certain pattern, or profile of traffic destinations. Maybe I find that 2% to Cox 3% to Shaw 2% to Comcast 5% to Roadrunner 2% to Adelphia and the next top 20 ASes represent the next 10% of traffic. Anonymized, this "Peering Profile" for Internet Gaming companies can probably be applied to other Internet Gaming companies and can provide a rough idea of good targets for peering and how much traffic can be expected at a peering point, as a percentage of their total traffic. Empirically, these top traffic destinations and volumes have been large enough, 10's of Mbps each, generally more than enough to justify peering a an IX where the breakeven point is 10-30Mbps. The design of the tool/template is pretty obvious from there. Side Note: See all the trouble we go through because traffic flow measurement is still non-trivial? If the netflow data is available at ingress/egress points, I was pointed to http://ehnt.sourceforge.net/ as a good freeware tool for evaluating and translating the netflow raw data. Bill
Re: Alternative to NetFlow for Measuring Traffic flows
At 09:16 PM 12/16/2002 -0500, K. Scott Bethke wrote: Impressive numbers but of course, slackers aside, if it was your connection and resources wouldnt you want more accurate information than just a guess? Yes, but I am also sympathetic to the challenges to ISPs in this economy, and the challenges with large networks where there are so many ingress/egress points that getting sampling in place is problematic. I hear from some Tier 1 ISPs that in some cases sampling is not available on the too new or too old NIC. In some cases there are simply too many points to measure, requiring too much disk, time, processing, etc. I heard stories of those that process the data monthly and do so at great expense, with the occasional crashes of the weekend jobs. Sometimes the quick and dirty approach is easier. Doing the research it was surprising to find how many of the largest ISPs in the world don't/can't do the detailed traffic analysis. > Interesting idea. Comments? Again it seems to iffy. What if you get a short DOS when you shift an ASN.. how much of a chump will you look like when you need that peer to be 1gbps and you hook up and its only pulling 2mpbs ? Good point - another assumption (3) that the traffic is normal predictable sinusoidal pattern such that the peak for the target AS matches the peak of the rest of the traffic. > The other approach some ISPs use is to set up a "trial" peering session, > usually using a private cross connect to measure the traffic volume and > relative traffic ratios. Then both side can get an idea of the traffic > before engaging in a contractual Settlement-Free Peering relationship. I like this one the best if I didnt have Netflow stat's... however I doubt everyone will allow this because of time, money, resources, security, etc. Yes, the Empirical Approach is most accurate but, besides the cost of implementing the trial peering, there are examples of Tier 2 ISPs trying to game the trial with a Tier 1 ISP in order to obtain the peering relationship. I heard stories of some pretty wacky routing and traffic engineering in order to demonstrate during the trial that ratios and traffic volumes fell within a certain range. ( The "Art of Peering" documents a few of these tactics.) I can understand why the Tier 1's are hesitant to do the trial peering even when they don't have the data to refute the "peering worthiness". I tend to look at peering as something you need to know when to do because the data tells you so. In this industry as it stands now why would you NOT run netflow stats to give you this information? all you are doing is wasting more money paying for transit that could be offloaded to peering. Me too, but differentiate between Tier 1 and Tier 2 solely for the motives; Tier 2's want to peer broadly to reduce transit fees, while Tier 1's by definition don't pay transit fees to anyone. And the flipside is also true.. why even worry about peering if you cant get more than a meg or two max to each AS? I have found peering to have additive value; a lot of 1-2 Mbps peering sessions can save as much money for you as a single large traffic peer. The more traffic, the stronger the case for peering. Bill
Alternative to NetFlow for Measuring Traffic flows
Hi all - Here is the problem: Everyone wants to know how much traffic would ultimately be passed in peering relationships at an IX before signing up/building into an IX. I heard an interesting solution recently to estimating the traffic volume destined to an AS in the absence of NetFlow or the like. The ability to measure traffic via sampling has been difficult for a variety of reasons (lack of staff resources, capabilities of the interface cards, expensive SW, etc.) and I found from talking to Peering Coordinators that less than 1 in 20 ISPs actually do the traffic measurements. In the absence of data, ISPs are often left to intuition, guessing that a particular AS would be a good peering candidate. Here is the Solution: Assuming that: 1) You are multi-homed 2) You have some ideas of who you would want to peer with 3) 1) You adjust routing to prefer one transit provider or the other for the AS 2) Shift traffic to the particular AS from one transit provider to the other, noting the change in the loads on the transit providers. If you do this at peak time you can get a rough estimate of the peak traffic to this AS, and therefore a rough order of magnitude estimate of the amount of traffic that would go to this AS in a peering relationship. (Rough Estimate means determining if the traffic volume is likely to be 2 Mbps vs. 20 Mbps vs. 200Mbps) Interesting idea. Comments? The other approach some ISPs use is to set up a "trial" peering session, usually using a private cross connect to measure the traffic volume and relative traffic ratios. Then both side can get an idea of the traffic before engaging in a contractual Settlement-Free Peering relationship. Bill
Re: latest variety of Nigeria scam
Just curious...Why post this - is there something unique here? I've been collecting these for years now, kinda like collecting insects. Currently I have about 120 variants from Princesses, widows, Prime Ministers, Ambassadors, and sons of deposed Kings, each with millions to give me. This looks like a plain old vanilla variety that we've all seen. Seems similar to spammers in terms of problem profile. A systematic approach is probably the right way to address it. (Please don't start a Spam thread here.) Bill At 12:50 AM 12/7/2002 +0100, hostmaster wrote: For those of you interested in the latest variety of the Nigeria -scam... it came straight from 217.78.73.1 SIOTEL NIGERIA LIMITED 217.78.73.5 SIOTEL NIGERIA LIMITED 217.78.73.160 SIOTEL NIGERIA LIMITED And for law enforcement in The Netherlands, this hopefully will lead you to something. best Bert === Dear x , WELTLOTTO-FIRMA WORLDLOTTO 41132, NL-1007 DB AMSTERDAM, THE NETHERLANDS. FROM: THE DESK OF THE DIRECTOR PROMOTIONS, INTERNATIONAL PROMOTIONS/PRIZE AWARD DEPARTMENT, REF: WFL/67-C337209635 ATTENTION ENTRANT: AWARD NOTIFICATION; FINAL NOTICE We are pleased to inform you of the announcement today, 13TH November 2002,of winners of the WELTLOTTO-FIRMA WORLDLOTTO/INTERNATIONAL PROGRAMS held on the 8TH OCTOBER, 2002. Your company, attached to ticket number 013-2316-2002-477, with serial number A025-09 drew the lucky numbers 37-13-34-85-56-42, and consequently won in category C. You have therefore been approved for a lump sum pay out of US$1,500,000.00 in cash credited to file REF NO. REF: WFL/67-C337209635. This is from total prize money of US$22,500,000.00 shared among the fifteen international winners in the category C. All participants were selected through a computer ballot system drawn from 30,000 names from Australia, New Zealand, America, Asia, Europe and North America as part our International Promotions Program, which is conducted annually. CONGRATULATIONS! Your fund is now deposited with a Finance and Security House insured in your name. Due to the mix up of some numbers and names, we ask that you keep this award strictly from public notice until your claim has been processed and your money remitted to your account. This is part of our security protocol to avoid double claiming or unscrupulous acts by participants of this program. We hope with a part of you prize, you will participate in our end of year high stakes US$1.3 billion International lotto. To collect your claim, please contact your claims officer immediately: MAXWELL FRIEDEL, FOREIGN SERVICE MANAGER, EUROSECURITIES NL, FAX : 31 205248020 EMAIL : [EMAIL PROTECTED] WEB URL : http:/www.eurosecurities-bv.com For due processing and remittance of your prize money to a designated account of your choice. Remember, you must contact your claims officer not later than DECEMBER 17TH, 2002. After this date, all funds will be returned as Unclaimed. NOTE: In order to avoid unnecessary delays and complications, please remember to quote your reference number in every one of your correspondences with your claims officer. Furthermore,should there be any change of your address, do inform your claims officer as soon as possible. _ **NB** QUOTE YOUR REFERENCE NUMBER AS THE SUBJECT OF YOUR MAIL, AND ATTACH THIS ONE WHEN YOU MAIL YOUR CLAIMS AGENT TO EXPEDITE YOUR CLAIM AND AVOID SERIOUS DELAYS. _ Congratulations again from all our staff and thank you for being part of our promotions program. Sincerely, THE DIRECTOR PROMOTIONS, WELT LOTTO FIRMA BV. www.weltlottofirma.s5.com N.B. Any breach of confidentiality on the part of the winners will result in disqualification. Please do not reply this e-mail.
Re: Cogent service
At 10:31 AM 9/20/2002 -0400, David Diaz wrote: > The only negative routing comments Ive heard are complaints about extra > hop counts. Dave - I know you know this and you are referring to an issue that both of us have heard The hidden assumption here is that the extra hops implies worse performance. This is perception rather than real. One could quite easily put in place a VPN or MPLS substrate and make all destinations appear "one hop away" without changing the underlying technology or performance of the network. A network application with clear latency/jitter/packet loss characteristics would be a more effective way to evaluate network fitness. I suspect what really happens is a) the is a performance problem somewhere in the path b) a traceroute is done c) the traceroute is misinterpreted - "the problem is packets go all over the place!" d) the misinterpretation is generalized to "more hops is bad" from what I've seen anyway. Bill
Re: Baltimore train tunnels (was Re: Vulnerbilities of Interconnection)
At 09:47 PM 9/7/2002 -0400, Sean Donelan wrote: >Unlike phone calls, TCP traffic doesn't occur in fixed bandwidth >increments. TCP traffic, 90% of Internet traffic, is elastic. By design, >TCP adjusts the traffic rate to keep the bottleneck congested. As the >bottleneck moves, traffic reacts by increasing or decreasing the rate to >match the available capacity. This feedback occurs independently of what >is happening on nearby traffic paths. Even if there is available >capacity on elsewhere, the current Internet design is not very good at >using it. Some people view this as an inefficient use of available >capacity, other people view it as a self-protective mechanism. Thank Goodness for well-behaved applications, right? ( Misbehaving TCP stacks and UDP-based apps don't obey these back off rules. ) I remember Van Jacobson gave a presentation back in 1997 that spoke about the problems with applications that didn't exhibit these characteristics: http://www.academ.com/nanog/october1997/ It would be interesting to see some recent verification that well-behaved TCP-apps are the norm on the Internet...any data out there in this regard? Bill
Re: Vulnerbilities of Interconnection
At 02:45 PM 9/5/2002 -0400, [EMAIL PROTECTED] wrote: >This obviously would be a thesis of Equinix and other collo space providers, >since this is exactly the service that they provide. It won't, hower, be a >thesis of any major network that either already has a lot of infrastructure >in place or has to be a network that is supposed to survive a physical >attack. Actually, the underlying assumption of this paper is that major networks already have a large global backbone that need to interconnect in n-regions. The choice between Direct Circuits and Colo-based cross connects is discussed and documented with costs and tradeoffs. Surviving a major attack was not the focus of the paper...but... When I did this research I asked ISPs how many Exchange Points they felt were needed in a region. Many said one was sufficient, that they were resilient across multiple exchange points and transit relationships, and preferred to engineer their own diversity separate from regional exchanges. A bunch said that two was the right number, each with different operating procedures, geographic locations, providers of fiber, etc. , as different as possible. Folks seemed unanimous about there not being more than two IXes in a region, that to do so would splinter the peering population. Bill Woodcock was the exception to this last claim, positing (paraphrasing) that peering is an local routing optimization and that many inexpensive (relatively insecured) IXes are acceptable. The loss of any one simply removes the local routing optimization and that transit is always an alternative for that traffic. > > > A couple physical security considerations came out of that research: > > 1) Consider that man holes are not always secured, providing access to > > metro fiber runs, while there is generally greater security within > > colocation environments > >This is all great, except that the same metro fiber runs are used to get >carriers into the super-secure facility, and, since neither those who >originate information, nor those who ultimately consume the information are >located completely within facility, you still have the same problem. If we >add to it that the diverse fibers tend to aggregate in the basement of the >building that houses the facility, multiple carriers use the same manholes >for their diverse fiber and so on. Fine - we both agree that no transport provider is entirely protected from physical tampering if its fiber travels through insecure passageways. Note that some transport capacity into an IX doesn't necessarily travel along the same path as the metro providers, particularly those IXes located outside a metro region. There are also a multitude of paths, proportional to the # of providers still around in the metro area, that provide alternative paths into the IX. Within an IX therefore is a concentration of alternative providers, and these alternative providers can be used as needed in the event of a path cut. > > 2) It is faster to repair physical disruptions at fewer points, leveraging > > cutovers to alternative providers present in the collocation IX model, as > > opposed to the Direct Circuit model where provisioning additional > > capacities to many end points may take days or months. > >This again is great in theory, unless you are talking about someone who >is planning on taking out the IX not accidently, but deliberately. To >illustrate this, one just needs to recall the infamous fiber cut in McLean >in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but >somehow let a cement truck to pour cement into Verizon's manhole that was >used by Level(3) and Worldcom. Terrorists in cement trucks? Again, it seems more likely and more technically effective to attack internally than physically. Focus again here on the cost/benefit analysis from both the provider and disrupter perspective and you will see what I mean. >Alex
Re: Vulnerbilities of Interconnection
At 12:44 PM 9/5/2002 -0400, [EMAIL PROTECTED] wrote: > One part that >we are looking at are the vulnerbilites of interconnection facilites. A quick point...Several folks have postulated that the internal (non-physical) threat dwarfs that of the physical threat, due to the lack of visibility, the difficulty of tracking and coordinating a response, and the millions of vulnerable systems world-wide capable of launching an internal attack. A physical attack (a hole in a wall for example) can typically be detected and corrected in a matter of hours or days, while an effective internal attack could be varied in time and scope causing at least as much damage invisibly for a much longer period of time. That said, a few years back I wrote the "Interconnection Strategies for ISPs" white paper, which speaks to the economics of peering using exchange points vs. using pt-to-pt circuits. It documents a clear break even point where large capacity circuits (or dark fiber loops) into an IX with fiber cross connects within a building are a better fit (financially) than pt-to-pt circuits. A couple physical security considerations came out of that research: 1) Consider that man holes are not always secured, providing access to metro fiber runs, while there is generally greater security within colocation environments 2) It is faster to repair physical disruptions at fewer points, leveraging cutovers to alternative providers present in the collocation IX model, as opposed to the Direct Circuit model where provisioning additional capacities to many end points may take days or months. Finally, I have seen a balancing act between how much it costs to protect against a disruption versus the cost of the disruption. In today's economy (unlike say a few years ago) more folks seem to be focused on doing this mathematically calculation rather than just picking full mesh interconnect topologies. Bill --------------- William B. Norton <[EMAIL PROTECTED]> 650.315.8635 Co-Founder and Chief Technical Liaison Equinix, Inc. Yahoo Instant Messenger ID: WilliamBNorton
Re: Do ATM-based Exchange Points make sense anymore?
At 01:13 PM 8/9/2002 -0700, Bill Woodcock wrote: > > Personally, I don't believe that ATM is 'bad' for > > shared-fabric exchange point. I mean, it works, and solves several > > problems quite easy: a) it's easily distributed via SONET services to > > folks who are not next to the ATM switch, b) it makes interconnection > > between networks safer (ie, not dealing with broadcast issues on a > > ethernet nap), c) virtual PI connections are easily accomplished, > d) there > > are varying degrees of interconnection speed (agreeably, less > important), > >All of the above are true of frame relay as well, which has the additional >benefit of not being funamentally incompatible with data networking. :-) You guys might find this interesting I'd like to share the more common "Religious debate points" regarding ATM-based vs. Ethernet-based IXes that I heard during the walk throughs (about 50 so far) of this paper (v1.6): -- ATM Advocate: First off, make sure you mention that ATM solves the key problem with "Broadcast Domain Internet Exchanges". Broadcast Domain Issues refers to problems when all ISP attachments are on the same Ethernet segment. Broadcast storms and other anomalies caused by one attached customer can adversely affects all others. Ethernet-based IX operators try and solve this problem administratively with MOUs Memorandum of Understanding (see the Appendix of the http://www.linx.net/joining/mou.thtml for an example of this) but it is solved in ATM by the private nature of PVCs, yielding a more stable peering infrastructure. Ethernet Advocate: Ethernet-based IXes can address these "Broadcast Domain" issues technically via private VLANs and direct cross connects between members. The "nature" of ATM as you describe it has a high overhead associated with it, specifically, by statically allocating bandwidth to a peering session with a historically spiky cyclical traffic characteristic. This static allocation of bandwidth prevents the multiplexing benefits of aggregation of lots of peering traffic sources. In addition to Ethernet-IXes being generally less expensive than ATM-based IXes, Ethernet interfaces are generally less expensive. This reduces the total peering costs, breakeven points, and increases the attractiveness of the Ethernet-based IX and therefore its likelihood of succeeding. Finally, Ethernet is what ISPs know, it is what they love. This ubiquity of and familiarity with Ethernet reduces the costs of operation, since operations folks only need to know how the Ethernet stuff works. ATM Advocate: Most Ethernet-based IXes primarily use the default LAN and are therefore subject to the "Broadcast Domain" issues. Ethernet may be less expensive but it is not shaping the traffic as ATM does. You pay for that functionality and stability. As for the operations argument, naturally one technology is easier to support than multiple technologies. This isn't a specific fault of ATM or Ethernet but rather and indication that choosing one and sticking with one is advantageous. Ethernet Advocate: In the Ethernet-based IX model, private cross connects allow one to scale beyond OC-12, the maximum reasonable capacity of ATM. The OC-48 ATM cards cost $250K, making it unreasonably expensive for the exchange of Internet Peering traffic. At the same time, the Ethernet-based model allows collocated folks to interconnect at Gigabit Ethernet, 10-GigE, whatever the peers choose. These direct connects are typically not allowed (for policy reasons) to occur at collocated ATM IXes. And PVCs are not the same things as a PNI as there are active electronics in the middle, some thing that can break, obscure troubleshooting, and limit flexibility with respect to interconnect types. Interesting points, and although orthogonal to the analysis in "Do ATM-based Internet Exchange Points Make Sense Anymore?", I am including these in the appendix to show these alternate views of the world. Am I missing any of the major (fact-based) views? Bill
Re: Do ATM-based Exchange Points make sense anymore?
Hi all - Thanks for all the feedback and keep it coming ! I'll summarize the 80 or so responses so far. As an aside, I especially liked this paper request: "I'd like to see a copy of your paper - please fragment it into 48 byte chunks." A couple points seem to come up from a bunch of folks: 1) Several folks said that they have seen transit prices at sub-$100/Mbps prices, some claiming the transit price quotes group around $75/Mbps. While the lower transit price points do strengthen the paper's argument, I would point out: a) there is a qualitative difference between transit providers, b) from my conversations there were higher and lower quotes than my $125-$100/Mbps, (A couple of people told me they were paying $350/Mbps, but they were at the tail end of a 3-year old contract that was signed when $350/Mbps was a great deal!) c) terms vary and location varies (rural guys are out of luck with no price competition, and some markets like Dallas are still high), d) I want to make sure that the reference transit price points in the Peering Model are representative of what is seen in the field. The bottom line is that I'm pretty comfortable with these numbers; $125/Mbps seems to be a price point that people can accept as a reference point for the Peering Analysis. And I've included the spreadsheet in the Appendix so you can adjust the transit price points as you see fit. 2) I explicitly mentioned in the paper that I ignored the equipment costs, in particular the OC-x POS and ATM interface cards and the equipment that ISPs would place in the Ethernet-based IX. This was because of the difficulty in determining a reference configuration (Juniper/Cisco, what series, new or used?), the price (people shared that 30% is easy to get) for a reference platform and then the lease term or amortization schedule. Some said depreciate things over 18 months, most said 24-36 months was the norm. In the past I have punted on this equipment question, but enough people mentioned it as a hole in the analysis (and a benefit of the ATM peering model) that if possible I'd like to include it into the analysis. So I guess I am asking for a base level reference configuration and price point that includes two router configurations for the peering model: 1) entry level router with an OC-3 card and FastE card to peer across an ethernet IX, and 2) next level router with an OC-12 card and GigE card to peer across a gigE IX I would also need an OC-3 ATM and OC-12 ATM price point. Round numbers are fine here as I'm looking for some reasonable number to plug in for equipment costs, knowing full well that everyones configuration will be different, and the spreadsheet will allow people to adjust the numbers to their situation. 3) Finally, several have pointed out that the decision about peering at an ATM fabric is not always a financial one. These were most common non-financial motivations I heard were: -) Performance: "I need to peer with this ISP regardless of the cost of that peering traffic." -) Contract Term: "We are in the middle of an n-year contract so we are stuck with the economics." (One ISP lost a peering session when the target ISP left, and is now left hanging in the wind with a fraction of their peering traffic to justify their peering. Moral: Before signing up with any IX, Make sure your target peers are not planning on moving out!) -) Perception: "To be a 'player' you have to be at xxx-IX." -) Let sleeping dogs lie: "If I ask my peer to change the peering session in any way, I fear they will use the opportunity to force us to re-qualify for peering." Most common was: -) Mathematics: "We haven't run the numbers like this yet. Didn't realize the unit costs here." Bill
Re: Do ATM-based Exchange Points make sense anymore?
Hi all - I have walked about 30 people through the "Do ATM-based Internet Exchange Points make sense anymore?" white paper and have received some really good feedback, suggestions and price points to calibrate the Peering Financial Model. I have applied these calibrations and I am ready to release the paper for wider review, but I'd like to share first the assumptions and calibration points for the model along with a few of the more interesting observations. The Business Case for Peering at an ATM-based Internet Exchange Point Peering looks pretty dismal in todays market. As I mentioned in an earlier message, the dominant issue is that transit and transport have dropped dramatically,while the cost of ATM-based peering has not dropped in kind. In todays market (from quotes shared with me) we see: Assumptions and Calibration Points -- Transit $125/Mbps with 500Mbps commit, $100/Mbps with 1000 Mbps commit. Transport (DC-ASH) $2500/mo for OC-3, $5000/mo for OC-12 Eth-IX fees: $2500/mo for 1/2 rack and FastE, $5000 for 1/2 rack and GigE Eth Framing Overhead: 6% HDLC Overhead: 4% ATM-IX fees: $11,000/mo for OC-3, and $26,000 for OC-12 transport and Port ATM cell tax: %20 Effective Peering Bandwidth=75% average utilization of available bandwidth (this means we assume that ISPs (for policy reasons) upgrade the peering infrastructure when the average utilization is 75%) These numbers are empirical and based on averages from the Internet Operations Community. The paper footnotes the sources. Observations When these numbers are plugged into the Peering Financial Models, we see that OC-3 ATM-based peering is "Effective" (less expensive than transit) for the very narrow range of 88Mbps-90Mbps. If an ISP can't send at least 88Mbps over the OC-3 to the ATM-IX, it would save money by simply buying transit. At 90 Mbps the OC-3 ATM must be upgraded. This narrow range leads me to believe that OC-3 ATM peering is simply not cost effective. Under the same assumptions (OC-3 into FastE IX), the Fast Ethernet-based Effective Peering Range is 40Mbps-70Mbps, a more reasonable range for medium scale peering. Applying the model to the ATM-OC-12 we see the Peering Breakeven Point is 260Mbps; if you don't send at least 260Mbps to the peering population then you should prefer simply to purchase transit. This peering infrastructure scales to 375 Mbps at which time it must be upgraded. In this Effective Peering Range the cost of traffic exchange ranges from $100/Mbps down to $69/Mbps when the Effective Peering Bandwidth is fully utilized. The same analysis applied to Gigabit Ethernet shows a much lower Peering Breakeven Point (100Mbps) with a broader range, scaling up to 448Mbps before the OC-12 must be upgraded, at which point the cost of peering traffic exchange is $22/Mbps. The bottom line is that the cost of the ATM Peering infrastructure, and the dropping price of transit and transport, have conspired to destroy the value proposition of ATM-based Internet Exchanges. Ethernet-based IXes are less expensive and have a broader "useful life", defined in this paper as "Effective Peering Range." As I walked folks through this paper I got the sense that most folks had not done this analysis and we opened some eyes here. Thanks to those who provided the empirical HDLC, ATM, and ethernet overhead figures. Including these provides a more fair comparison between ATM and Ethernet-based IXes. If you would like a copy of the paper please send e-mail to [EMAIL PROTECTED] and I'd be glad to send you a copy. As always, I'd love to hear your feedback; that is how these papers become valuable resources for the community. Thanks - Bill
Re: Do ATM-based Exchange Points make sense anymore?
>Can you, please, explain why you didn't consider Frame Relay >based exchange in your analysis? I don't have much insight into Frame Relay-based Internet Exchange Points ;-) The majority of IXes around the world are ethernet-based, with some legacy FDDI and a few ATM IXes. It is in these areas that I have done the most data collection. The same analysis could be applied to peering across WANs and MANs as compared with buying transit though. It might be interesting provided I can get some market prices for transport and ports. Why look at ATM? Right now almost everyone I am speaking with is seeing massive drops in transit and transport prices, even below the points I quoted, but with no comparable price drop in ATM ports or transport into an ATM cloud. These forces lead to a point where a connection to an ATM IX makes no sense (from a strictly financial standpoint). I have another 10 folks to walk through the paper to make sure I'm not missing anything in the analysis, and I'll post to the list when the paper is available. If you are interested I'd love to walk you through it to get your take. One point a couple other folks brought up during the review (paraphrasing) "You can't talk about a 20% ATM cell tax on the ATM-based IX side without counting the HDLC Framing Overhead (4%) for the OC-x circuit into an ethernet-based IX." Since the "Effective Peering Bandwidth" is the max peering that can be done across the peering infrastructure, this is a good point and has now been factored into the model and analysis. Bill
Last Call: NANOG Peering BOF
Hi All - At the Peering BOF Monday evening, so far I have Peering Coordinators lined up from Adelphia, CableVision, Comcast, Cox, DACOM Korea, Equinix, GNAPs, ICG, ISC, Japan Telecom, Merit, Powered, Shaw, TELUS, T-Systems, Videotron, Yahoo! and C&W. We can take another 5-7 folks on the agenda, so... If you are a peering coordinator and would like to introduce yourself to the other Peering Coordinators, please fill out and e-mail the RSVP form below to [EMAIL PROTECTED] We had a lot of walk-ons last time and we will do that again this time, but if you RSVP I will have your contact info and RSVP info in the screen behind you as you introduce yourself, so it works better if you RSVP. A couple suggestions came from the field as well: Bring plenty of cards, network maps showing relevant info (peering points, backbone topology and size of pipes, etc). After the BOF we will break to the bar where the real work happens ;-) See you in Richmond Hill ! Bill - Previous Call for Participants -- Hi all - NANOG is only three weeks away and Monday evening at NANOG there will be another Peering BOF ; thanks to those that suggested this on the survey forms! We'll do this the same way as last time / the same way the Peering Personals ran at the last GPF: *Peering Coordinators*: Send me the completed RSVP form below. I'll assemble these into logos, icons, AS#s and contact info With this backdrop, each of you in turn get a chance to stand up and a) introduce yourself, your network, b) what you are looking for in a peer, c) why folks should want to peer with you, and d) which locations you currently or plan to peer. Making the initial contact with the potential peer is (oddly enough) the most difficult parts of peering, and the Peering Personals has proven to be an effective (and lively!) way to make those initial contacts. So *Peering Coordinators* - send me those RSVPs ! Since we only have 90 minutes I'm going to limit the number of Peering Coordinators to 25 or so. If there is time remaining we'll use the rest of the time for ad hoc Peering Personals as we did last time. A couple comments: I noticed on the thread "Interconnects" folks were talking about willingness to peer and MLPAs. At least from the conversations I had during my research on Peering, I found relatively little interest in MLPAs. For those using contracts for peering, folks preferred to control peering using their own contracts written by their lawyers, stating their evolving peering terms and conditions, and generally felt somewhat like they were losing control by signing up to a MLPA document. At the same time, I have found from running these Peering Personals and talking with these Peering Coordinators, that maybe 80% of all Peering Coordinators had a relatively open peering policy. By "Relatively Open" I mean that they would peer in any single location or multiple location with companies that they would not consider to be a prospective customer. This openness was surprising given all the huff and puff on mailing lists over the years about *not* being able to get peering. We'll see if my 80% figure rings true at the Peering BOF, and I'll share a couple anecdotes about an emerging set of significant traffic open peers at the Peering BOF. Bill -- RSVP FORM -- Clip Here --- Please Fill out and e-mail to [EMAIL PROTECTED] with Subject: Peering BOF V Name: __ Email: __ Title: __ Company: ___ AS#(s): _ Check each that applies: ___ We are an ISP (sell access to the Internet) -- OR -- ___ We are a Non-ISP (content company, etc.) ___ We are Content-Heavy -- OR -- ___ We are Access-Heavy ___ We generally require peering in multiple locations -- OR -- ___ We will peer with anyone in any single location ___Peering with Content Players or Content Heavy ISPs is OK by us ___ We have huge volumes of traffic (lots of users and/or lots of content) (Huge: > 1 Gbps total outbound traffic to peers and transit providers) ___ We have a global network ___ We require written contracts for peering ___ We have a U.S. Nation-Wide Backbone (East Coast, West Coast, and at least one location in the middle) --- snip
Peering BOF V - Call for Participants
Hi all - NANOG is only three weeks away and Monday evening at NANOG there will be another Peering BOF ; thanks to those that suggested this on the survey forms! We'll do this the same way as last time / the same way the Peering Personals ran at the last GPF: *Peering Coordinators*: Send me the completed RSVP form below. I'll assemble these into logos, icons, AS#s and contact info With this backdrop, each of you in turn get a chance to stand up and a) introduce yourself, your network, b) what you are looking for in a peer, c) why folks should want to peer with you, and d) which locations you currently or plan to peer. Making the initial contact with the potential peer is (oddly enough) the most difficult parts of peering, and the Peering Personals has proven to be an effective (and lively!) way to make those initial contacts. So *Peering Coordinators* - send me those RSVPs ! Since we only have 90 minutes I'm going to limit the number of Peering Coordinators to 25 or so. If there is time remaining we'll use the rest of the time for ad hoc Peering Personals as we did last time. A couple comments: I noticed on the thread "Interconnects" folks were talking about willingness to peer and MLPAs. At least from the conversations I had during my research on Peering, I found relatively little interest in MLPAs. For those using contracts for peering, folks preferred to control peering using their own contracts written by their lawyers, stating their evolving peering terms and conditions, and generally felt somewhat like they were losing control by signing up to a MLPA document. At the same time, I have found from running these Peering Personals and talking with these Peering Coordinators, that maybe 80% of all Peering Coordinators had a relatively open peering policy. By "Relatively Open" I mean that they would peer in any single location or multiple location with companies that they would not consider to be a prospective customer. This openness was surprising given all the huff and puff on mailing lists over the years about *not* being able to get peering. We'll see if my 80% figure rings true at the Peering BOF, and I'll share a couple anecdotes about an emerging set of significant traffic open peers at the Peering BOF. Bill -- RSVP FORM -- Clip Here --- Please Fill out and e-mail to [EMAIL PROTECTED] with Subject: Peering BOF V Name: __ Email: __ Title: __ Company: ___ AS#(s): _ Check each that applies: ___ We are an ISP (sell access to the Internet) -- OR -- ___ We are a Non-ISP (content company, etc.) ___ We are Content-Heavy -- OR -- ___ We are Access-Heavy ___ We generally require peering in multiple locations -- OR -- ___ We will peer with anyone in any single location ___Peering with Content Players or Content Heavy ISPs is OK by us ___ We have huge volumes of traffic (lots of users and/or lots of content) (Huge: > 1 Gbps total outbound traffic to peers and transit providers) ___ We have a global network ___ We require written contracts for peering ___ We have a U.S. Nation-Wide Backbone (East Coast, West Coast, and at least one location in the middle) --- snip
The Art of Peering : The Peering Playbook
Hi all - Folks were talking about Traffic Ratios, Depeering, etc. that reminded me I should probably thank everyone for contributing to the "Tactical Peering" white paper which has now been renamed "The Art of Peering : The Peering Playbook". Thanks to the feedback from folks on this list and at RIPE and the Gigabit Peering Forum I have released version 1.0 of this document and it is available to anyone who would like a copy. Send me e-mail at [EMAIL PROTECTED] with the Subject: Art of Peering and I'll send it back directly, or alternatively you can get it from the Equinix web site. In this paper I asked the Peering Coordinators the question "What do you do if noone answers your peering request at peering@.net ? What are the 'Tricks of the Trade' that distinguish seasoned Peering Coordinators from newbies?" The Summary (below) does the best job of highlighting the techniques detailed in the paper: Summary We have presented 19 peering maneuvers that the Peering Coordinator Community have effectively used to obtain peering. 1) The Direct Approach uses peering@.net , phone calls, face to face meetings, or some such direct interaction to establish peering. 2) The Transit with Peering Migration tactic leverages an internal advocate to buy transit with a contractual migration to peering at a later time. 3) The End Run Tactic minimizes the need for transit by enticing a direct relationship with the target ISP's largest traffic volume customers. 4) In Europe the Dual Transit/Peering separates the peering traffic from the transit traffic using separate interface cards and/or routers. 5) Purchasing Transit Only from Large Tier 2 ISPs is an approach to reduce the risk of being a customer of a potential peer on the road to Tier 1 status. 6) Paid Peering as a maneuver is positioned by some as a stepping stone to peering for those who don't immediately meet the peering prerequisites. 7) In the Partial Transit tactic, the routes learned at an exchange point are exchanged with the peer for a price slightly higher than transport costs. 8) The Chicken tactic involves de-peering in order to make the other peer adjust the peering relationship. 9) In the Traffic Manipulation tactic, ISPs or content players force traffic along the network path that makes peering appear more cost effective. 10) The Bluff maneuver is simply overstating future traffic volumes or performance issues to make peering appear more attractive. 11) The Wide Scale Open Peering Policy as a tactic signals to the Peering Coordinator Community the willingness to peer and therefore increases the likelihood of being contacted for peering by other ISPs. 12) The Massive Colo Build tactic seeks to meet the collocation prerequisites of as many ISPs as possible by building POPs into as many exchange points as possible. 13) The Aggressive Traffic Buildup tactic increases the traffic volume by large scale market and therefore traffic capture to make peering more attractive. 14) Friendship-based Peering leverages contacts in the industry to speed along and obtain peering where the process may not be in place for a peering. 15) The Spam Peering Requests tactic is a specific case of the Wide Scale Open Peering tactic using the exchange point contact lists to initiate peering. 16) Purchasing Legacy Peering provides an immediate set of peering partners. 17) The Bait and Switch tactic leverages a large corporate identity to obtain peering even though ultimately only a small subset or unrelated set of routes are actually announced. 18) The False Peering Outage tactic involves deceiving an ill-equipped NOC into believing a non-existing peering session is down. 19) The Leverage Broader Business Arrangement takes advantage of other aspects of the relationship between two companies to obtain peering in exchange for something else. Thanks again for your help! If there are questions or comments I'd love to hear them; I fully expect this document (like the other white papers) to evolve over time. Bill ----------- William B. Norton <[EMAIL PROTECTED]> 650.315.8635 Co-Founder and Chief Technical Liaison Equinix, Inc.
