Re: concern over public peering points [WAS: Peering point speed publicly available?]
let's just say that my experience is not all that reliable. i i suspect it varies greatly between colo/sub-switch providers. but considering the cost, i ain't got no complaints. qed. randy
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 01:39:03PM -0700, Randy Bush <[EMAIL PROTECTED]> wrote: > building owners. so, though the six does have a core, it is > also kinda splattered into switches all over the building; with > ease of connection and low cost being achieved at the expense > of reliability. Though that's true, the SIX has been extremely reliable: one unscheduled core outage in the last 3 years (about 30 minutes due to power loss). In one other case, an extension switch (7 peers) was disconnected for about 30 minutes to troubleshoot a potential problem. Peer-operated extension switches have also been very reliable. Most are above 99.9% availability including scheduled maintenance and 99.99% for unscheduled problems. The SIX's staffed 24x7 NOC lets peers treat it like any other carrier relationship, with one phone number to report a problem. Often the ops staff at national networks never know the SIX is non-profit or donation-supported. Peers of all sizes seem happy with the reliability. Everyone has open-posting mailing lists and an annual opportunity to elect the Board of Directors, so there is recourse if circumstances change. Cheers, Troy (SIX janitor)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: > but you seem to think they are served in exchange points, and not > just to those that run them, but to all comers. very cool. > > sad to say, we're past 1999 now. out here in the free world (and > those countries we bomb and/or invade[0]) folk seem to want us to > pay for what we eat. bummer, eh? The weird thing is that I (and partners) have been running an IX wth 4 nodes since 2001 with the business model I have mentioned and as far as I can calculate, we have at least made break-even. At $5k a year for FE and $10k a year for GE and letting the ISP provide their own access to the IX via whatever means they have available, it's possible to run an IX if you just want to provide the IX L2 unicast service and not have a lot of other services around. We calculated that we needed three customers per PoP and we've had more than that. The initial investment in switches was approx $50k per PoP. Running L2 switches is quite simple, I don't see what all the fuss is about. If the above model doesnt work in your area, though luck for you, guess you have to pass on the added cost to your customers. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
beware. six is funny. it's in seattle's carrier hotel, the westin, 32 floors of racks, more colo providers than fleas on a dawg, and very very low inter-suite fiber rates from the building owners. so, though the six does have a core, it is also kinda splattered into switches all over the building; with ease of connection and low cost being achieved at the expense of reliability. and costs are distributed along with the six infrastructure. so colo provider A may have a switch and charge $a to access it, while colo provider B may charge $b, where $b != $a. for a small local exchange this is ok, even cool. i would not want to do similarly in virginmania or palo attitude, and i would not join the six if i was a major player (only a research rack is on the six). my internal indirect costs would not be worth the traffic shed. randy
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 10:57:20AM -0700, Michael Smith wrote: > > At the Seattle Internet Exchange a, granted, smaller peering exchange, > you have to account for the following costs (and, mind you, this list is > not exhaustive). > > 1) 1 Rack > 2) Space for the rack in a secure facility > 3) AC for the equipment > 4) Power for the equipment (including line and UPS) > 5) Fiber and Copper runs to the facility for cross-connects > 6) Terminations of (5) > 7) O&M of space and gear > 8) Layer 8 and 9 negotiation of (1) through (7) to keep costs down. > > That's not a trivial set of expenses, particularly when there are > limitations in place to recovering costs via non-cash methods, such as > advertising the hosting of the exchange. > > Thankfully, there is some altruism on the behalf of several parties that > allow the exchange to continue providing "zero cost" connections to > participants. I hardly think the cost of their time and effort is > "marginal". Which means that SIX's costs would be completely covered by charging each member with a GigE port $1k/mo. I would rather pay them the $1k/mo with the expectation that they will be able obtain quality hardware (which btw doesn't necessarily mean running to their favorite vendor and asking for the most expensive product available), provide reliable service, handle growth, etc. I would not however, pay them $14k/mo for the same service. I count 68 active participants on the SIX website. I won't venture a guess as to how many have GigE ports, and a few are connected from PAIX, etc, but I would bet that there is more than enough business available to cover the costs of intelligent spending. You could probably still give away FastE ports for free, and pretty much assume that any major ISP who can afford the GigE port and sees value in connecting with the smaller guys will go ahead and pay for it. -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
>> i look forward to my next trip to sweden, where i expect many >> nice free lunches > If you start working in a resturant, you can probably expect that. but you seem to think they are served in exchange points, and not just to those that run them, but to all comers. very cool. sad to say, we're past 1999 now. out here in the free world (and those countries we bomb and/or invade[0]) folk seem to want us to pay for what we eat. bummer, eh? randy -- [0] - bumber sticker of the week "We're making enemies faster than we can kill them!"
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: > i look forward to my next trip to sweden, where i expect many > nice free lunches If you start working in a resturant, you can probably expect that. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
i look forward to my next trip to sweden, where i expect many nice free lunches randy
RE: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Michael Smith wrote: > 1) 1 Rack > 2) Space for the rack in a secure facility > 3) AC for the equipment > 4) Power for the equipment (including line and UPS) This can be had for approx $300-1000 a month in my market. > 5) Fiber and Copper runs to the facility for cross-connects > 6) Terminations of (5) This is carried on a per connection basis in my market. > 7) O&M of space and gear $50-100k over three years isn't that much. > 8) Layer 8 and 9 negotiation of (1) through (7) to keep costs down. I'd say that the time spent in negotiations is wasted, manpower is too expensive compared to the costs involved. > Thankfully, there is some altruism on the behalf of several parties that > allow the exchange to continue providing "zero cost" connections to > participants. I hardly think the cost of their time and effort is > "marginal". In the big picture it's marginal. Asking someone to patch a cable is a 10 minute job and the patch cable costs perhaps 30-50 dollars. Handling an invoice for this job is a major cost in the equation so yes, altruism is great. We gathered players that already had engineers, already had billing departments, already had all of the above you were referring to and get everybody to agree on a way to cooperate. The marginal costs for everybody to establish 5 PoPs and interconnecting them was quite low and since there are no billing being done between participants, that cuts down on paperwork as well. It's like a car pool. If everybody is going to bill everybody it's going to be a big operation. If you just agree to drive every fourth day and carry your own costs, everybody is better off. I realise from everybody who answered that we live in different markets and do things differently. I just think you're making it too advanced and that increases cost until public IXes stop to make sense. Keep it simple. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Stephen J. Wilcox wrote: > This is simply untrue. > > Whilst it is possible to establish an exchange with minimal cost if it is > successful your costs will soon escalate. > > To provide carrier class service for the worlds top carriers you need to invest > in the latest hardware, you need to house multiple switches and odfs in suites, > you need to pay a team of engineers to run the exchange 24x7, you need to > maintain vendor support agreements. IXes are not for "top carriers", they're for the small and middle players, and in some cases for the top players to talk to smaller players. IXes is a way to cheaply exchange traffic. It's better to establish two IX switches and run them with 99.9% availability than to have a single IX switch and aim for 99.999%. > If you're exchange is in an already developed location then my observation is > that you need to have the above if you are to attract the larger networks which > in turn brings in the traffic and noc requirements that see increasing costs. If you're already an operator or colo facility owner, you already have all of that, which makes the cost of running an IX much less than if you're a separate entity who have to set up all these facilities. I work in an environment where IXes are readily available in all major metropolitan areas where we are, and they don't cost an arm and a leg and fiber is cheap and readily available, so we try to establish everywhere. This brings the impact of a single IX being down to very negligable, so we definately don't need 99.999%. Off the top of my head, I'd estimate that the cost of being present at an exchange here is around $1-5k per gig per month (including router port, fiber connection and IX exchange fee). We run these at approx 50% utilisation so the price per megabit is $5-10/megabit per month. This also adds a lot of reduced latency from our customers to our competitors customers which is very appreciated, it also cuts down on long-haul costs. If an IX costs $50-100k a year for a gig it tilts the whole equation, so I can understand if a lot of people don't like them if that's the cost of being connected. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
RE: concern over public peering points [WAS: Peering point speed publicly available?]
> -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Mikael Abrahamsson > Sent: Saturday, July 03, 2004 10:22 AM > To: [EMAIL PROTECTED] > Subject: Re: concern over public peering points [WAS: Peering point speed > publicly available?] > > > On Sat, 3 Jul 2004, Laurence F. Sheldon, Jr. wrote: > > > Does the person that sweeps the floor do so for free? And supply the > > broom? > > The marginal cost of half a rack being occupied by an IX switch in a > multi-hundred-rack facility is negiglabe. Yes, it should carry a cost of a > few hundred dollars per month in "rent", and the depreciation of the > equipment is also a factor, but all-in-all these costs are not high and if > an IX point rakes in $200k a year that should well compensate for these > costs. > > -- > Mikael Abrahamssonemail: [EMAIL PROTECTED] > > At the Seattle Internet Exchange a, granted, smaller peering exchange, you have to account for the following costs (and, mind you, this list is not exhaustive). 1) 1 Rack 2) Space for the rack in a secure facility 3) AC for the equipment 4) Power for the equipment (including line and UPS) 5) Fiber and Copper runs to the facility for cross-connects 6) Terminations of (5) 7) O&M of space and gear 8) Layer 8 and 9 negotiation of (1) through (7) to keep costs down. That's not a trivial set of expenses, particularly when there are limitations in place to recovering costs via non-cash methods, such as advertising the hosting of the exchange. Thankfully, there is some altruism on the behalf of several parties that allow the exchange to continue providing "zero cost" connections to participants. I hardly think the cost of their time and effort is "marginal". Mike NoaNet
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Mikael Abrahamsson wrote: > > On Sat, 3 Jul 2004, Randy Bush wrote: > > > no. in the first case, you're just hiding the incremental costs. > > eventually, some bean counter is gonna want to recover them, and > > then folk get quite unhappy. > > What costs are you referring to? You basically need a few hours time per > month from engineers and billing department. This for an exchange that has > 20 ISPs connected to it. The amount of traffic isn't really a factor, but > the one I know of and am part of running carries multi-gigabit. This is simply untrue. Whilst it is possible to establish an exchange with minimal cost if it is successful your costs will soon escalate. To provide carrier class service for the worlds top carriers you need to invest in the latest hardware, you need to house multiple switches and odfs in suites, you need to pay a team of engineers to run the exchange 24x7, you need to maintain vendor support agreements. >From empirical data this cost is in the order of a few million dollars per year. This may not be a lot of money compared to the annual turnover of the large carriers but eg for a typical exchange $5m between 150 companies is on average about $3k/mo each (of course this will likely be skewed so that the top few companies pay more). If you're exchange is in an already developed location then my observation is that you need to have the above if you are to attract the larger networks which in turn brings in the traffic and noc requirements that see increasing costs. Steve
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 08:47:11AM -0700, Randy Bush wrote: > > >> The price being charged for the public exchange ports is > >> non-trivial > > Only at the (very few) commercial exchanges. The vast majority > > are free or of trivial expense. > > by count of small 10/100 switches or by traffic volume? > > it costs to build, maintain, and manage an exchange which carries > significant traffic. costs get recovered. life is simple. I tend to get suspicious when I know the exchange isn't charging enough money to cover its costs. I also don't see a need for a "free exchange" either. I'm perfectly willing to pay a fair price for the service, and I at least want the BELIEF that I am going to get a certain level of service from the exchange, not "but we can't afford..." or "duhhh?". It seems that most commercial network operators agree, as you rarely see them popping up at joe bob's local alternative new exchange point, even when it is free. The cost for the exchange hardware is really not that much. Just to throw out some numbers, you can snag a new 6509 w/SUP720 and 48-SFP GE for less than $50k with very modest discounts. Admittidly this is relatively new technology compared to most GE exchanges currently deployed, but the pricing a couple years ago was around the same for the Floundry's that everyone deployed, just at a lower density. A successful exchange probably has multiple switches and some 10GE trunks, but with a few customers paying industry average recurring fees this quickly pays for itself. The euro players are really the ones to look to for examples here, US players have been complete failures (especially with multi-site linked exchanges). The guys best positioned to do it are the actual colo operators who already have a technician staff on site, they really only need 1-2 higher level engineers, a support contract for when the switch crashes, etc. The real cost and value of an exchange point is the marketing (i.e. showing up at nanog and giving presentations about it, creating your own peering events, having sales folks promoting the product, etc), not the hardware. -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Laurence F. Sheldon, Jr. wrote: > Does the person that sweeps the floor do so for free? And supply the > broom? The marginal cost of half a rack being occupied by an IX switch in a multi-hundred-rack facility is negiglabe. Yes, it should carry a cost of a few hundred dollars per month in "rent", and the depreciation of the equipment is also a factor, but all-in-all these costs are not high and if an IX point rakes in $200k a year that should well compensate for these costs. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
Mikael Abrahamsson wrote: On Sat, 3 Jul 2004, Randy Bush wrote: no. in the first case, you're just hiding the incremental costs. eventually, some bean counter is gonna want to recover them, and then folk get quite unhappy. What costs are you referring to? You basically need a few hours time per month from engineers and billing department. This for an exchange that has 20 ISPs connected to it. The amount of traffic isn't really a factor, but the one I know of and am part of running carries multi-gigabit. Does the person that sweeps the floor do so for free? And supply the broom? -- Requiescas in pace o email Ex turpi causa non oritur actio http://members.cox.net/larrysheldon/
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: > no. in the first case, you're just hiding the incremental costs. > eventually, some bean counter is gonna want to recover them, and > then folk get quite unhappy. What costs are you referring to? You basically need a few hours time per month from engineers and billing department. This for an exchange that has 20 ISPs connected to it. The amount of traffic isn't really a factor, but the one I know of and am part of running carries multi-gigabit. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
> What is significant traffic? What is the cost? If you have an exchange > with let's say 20 people connected to it and they all connect using GE. > Running this exchange in an existing facility with existing people, you > can easily run it for under $10k per year per connected operator or less > as you already have engineers that are on site frequently, you already > have a billing department etc. > > It's when the exchange is being run by a separate entity that needs a > marketing department, a well-paid staff of managers, technicians etc that > price really goes up. All this to basically manage a simple ethernet > switch that needs some patching a couple of times a month at maximum. no. in the first case, you're just hiding the incremental costs. eventually, some bean counter is gonna want to recover them, and then folk get quite unhappy. and, there are known issues when a colo or transit provider is the exchange. [ note that i am not talking about small local friendly exchanges. i mean stuff that carries multi-gig. it's like is-is, almost no one runs it, only the few folk who carry most of the internet's traffic. ] randy, who contributes to and peers at the seattle internet exchange
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: > it costs to build, maintain, and manage an exchange which carries > significant traffic. costs get recovered. life is simple. What is significant traffic? What is the cost? If you have an exchange with let's say 20 people connected to it and they all connect using GE. Running this exchange in an existing facility with existing people, you can easily run it for under $10k per year per connected operator or less as you already have engineers that are on site frequently, you already have a billing department etc. It's when the exchange is being run by a separate entity that needs a marketing department, a well-paid staff of managers, technicians etc that price really goes up. All this to basically manage a simple ethernet switch that needs some patching a couple of times a month at maximum. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, ren wrote: > 5. Costs. Private peering is expensive, don't let anyone fool you. There > is a resource investment in human terms that is rarely calculated properly, I agree with you 100%. Working at a nordic european operator being present at LINX, AMSIX and all the northern europe exchanges my reasoning is this: With IXes you buy one highspeed interface and get lots of peers and you can peer with people you might only exchange a few megabit/s with. Buying loads and loads of OC3s, T3s, OC12 to peer with and purchasing fiber patching to interconnect these just doesnt make sense when you can buy a GE or 10GE interface and get tens or hundreds of peers on that single interface without re-patching or establishing any new fiber connections. We have a very liberal peering policy which makes peering a pure operational decision, being handled by the line organisation. Each peering takes approx 5-10 minutes of someones time and that's it. No meetings of peering coordinators or alike, so those people are freed up to do better things. In a lot of the european exchanges all graphs of all ports on the IX is available to you as a member (or even publically available). If someone runs their port full, you probably know about it. Playing the peering game and trying to increase cost for someone else means you increase your own cost as well. Is that worth it? You have to be pretty big to justify it... -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Who broke .org?
JW> Date: Sat, 3 Jul 2004 11:22:34 -0400 JW> From: Jeff Wasilko JW> On Sat, Jul 03, 2004 at 06:45:44AM -0700, Bill Woodcock wrote: JW> > JW> > Uh, how much additional down-time did you want? Rolling JW> > the clock back a decade isn't going to make things JW> > _better_. JW> JW> Why do you say that? JW> JW> .com and .net seem to work just fine without the extreme JW> reliance on 2 anycasted servers (i.e. they are serving up 13 JW> different NS records). "One anycast implementation is having trouble, therefore anycast must be inherently bad" is hardly good logic. Something I forgot to add to my anycast ramblings the other evening: Say one has ns1.domain.tld and ns2.domain.tld both anycasted. Assuming pods have two machines, set your MEDs[*] such that ns1 prefers server "A" and ns2 prefers server "B". This helps queries destined for different NSes hit different machines. [*] Or whatever knob you use. JW> I realize .com/.net may be using anycast as well, but they've JW> managed to engineer a solution that is stable. I don't think gtld-servers.net uses anycast; someone correct me if I'm wrong. F-root != gtld-servers.net. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: concern over public peering points [WAS: Peering point speed publicly available?]
>> The price being charged for the public exchange ports is >> non-trivial > Only at the (very few) commercial exchanges. The vast majority > are free or of trivial expense. by count of small 10/100 switches or by traffic volume? it costs to build, maintain, and manage an exchange which carries significant traffic. costs get recovered. life is simple. randy
Re: Who broke .org?
PGB> Date: Sat, 3 Jul 2004 11:28:10 +0100 PGB> From: Per Gregers Bilse PGB> At least the previous outage (a couple of weeks ago) had PGB> nothing to do with anycast, I was getting NXDOMAIN replies PGB> back, and no kind of fallback or non-anycast deployment PGB> would have helped. Moreover, it would be nice if a lookup test suite detected the incorrect NXDOMAIN responses and yanked the routes for the pod(s) in question. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: Who broke .org?
On Sat, Jul 03, 2004 at 06:45:44AM -0700, Bill Woodcock wrote: > > On Fri, 2 Jul 2004, Jeff Wasilko wrote: > > Can't we just go back to non-anycast, please? > > Uh, how much additional down-time did you want? Rolling the clock back a > decade isn't going to make things _better_. Why do you say that? .com and .net seem to work just fine without the extreme reliance on 2 anycasted servers (i.e. they are serving up 13 different NS records). I realize .com/.net may be using anycast as well, but they've managed to engineer a solution that is stable. .org was pretty reliable when it was being run by the same folks that are still running .com/.net. .org broke one month after it was moved to UltraDNS, and has since broken at least 4 times (based on reports to NANOG). How many times have there been significant outages in .com/.net in the past 10-11 months? Wouldn't there be a huge uproar (a-la sitefinder) if .com/.net were as unreliable as .org has been? -j (wishing his domain wasn't in .org anymore)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 08:28:50AM -0400, ren wrote: > At 02:07 AM 7/3/2004 -0400, Richard A Steenbergen wrote: > >b) The price being charged for the public exchange ports is non-trivial > > (especially compared to the cost of transit these days!), and is billed > > on a port basis instead of a usage basis (at least in the US). Since > > public peering is treated as a "necessary evil", with traffic moved to > > much more economical private peers when they start getting full, no one > > wants to provision extra capacity ahead of demand (in fact, in the US > > it is exceedingly rare to see anyone with 2 ports on a single public > > exchange). > > As one of the folks who gets questioned by Sales all the time > about the reasons behind the multiple shared fabric ports at the IXs I'll > gladly explain why we have 14 in the US at present and are preparing for > ~5-10 abroad. You're definitely one of the rare few, especially given your size. In Europe it seems far more common for people to provision multiple ports and make certain they have capacity. In the US, even the couple of other folks I can think of who actually decided to provision multiple ports on the "modern exchanges" we're thinking of ended up sitting with congestion for some number of weeks before they actually did it. The general line of thinking here is "ok exchange port is getting full, lets move someone big to a PNI". Are there even any exchange points in the US who are actually doing 10GE right now (major and production, not someone tinkering)? One way or another, there is definitely room for improvement in the technology of public peering. Then again, with some classic exchanges (that are still considered viable, aka not mae's, aads, pbnap, etc) still charging the same prices they were back in 1999, aka more than transit, perhaps there is room for improvement in the financial model as well. :) > 5. Costs. Private peering is expensive, don't let anyone fool you. There > is a resource investment in human terms that is rarely calculated properly, > all the way from planning of inventory to planning for capacity augments > after the physical install. It is often difficult to capture the cost to > roll all those fibers that are improperly installed. This I'm sure you are > painfully aware of . *grumble* Indeed. The one redeeming quality of your favorite overpriced colo and mine is that when they go to hook up a crossconnect they extend it all the way to the gear without a dozen more tickets, they manage to hook it up correctly the first time, without 1-2 hours of handholding or playing "find the port", and without the need to dispatch techs or pay for half an hour of remote hands to roll the damn fibers. :) -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Who broke .org?
On Fri, 2 Jul 2004, Jeff Wasilko wrote: > Can't we just go back to non-anycast, please? Uh, how much additional down-time did you want? Rolling the clock back a decade isn't going to make things _better_. -Bill
Re: concern over public peering points [WAS: Peering point speed publicly available?]
Bill Woodcock writes on 7/3/2004 7:02 PM: On Sat, 3 Jul 2004, Richard A Steenbergen wrote: > The price being charged for the public exchange ports is non-trivial Only at the (very few) commercial exchanges. The vast majority are free or of trivial expense. But some people really like to lose money, since then they get to hang out with VCs and feel like movers and shakers, rather than feeling like peons who have to actually turn a profit. Hah. http://www.nixi.org - described to a T. srs -- suresh ramasubramanian [EMAIL PROTECTED] gpg EDEDEFB9 manager, security and antispam operations, outblaze ltd
Re: ultradns reachability
On Fri, 2 Jul 2004, Stephen J. Wilcox wrote: > 10.1.0.1 Anycast1 (x50 boxes) > 10.2.0.1 Anycast2 (x50 boxes - different to anycast1) > In each scenario two systems have to fail to take out any one customer.. but > isnt the bottom one better for the usual pro anycast reasons? Correct, and that's what's done whenever engineering triumphs over marketing. The problem is that there's always a temptation to put instances of both clouds at a single physical location, but that's sabotaging yourself, since then the attack which takes down one will take down the other as well. With DNS, it really makes sense to do what you're suggesting, since DNS has its own internal load-balancing function, and having two separate clouds just means that you're giving both the anycast and the DNS client load-balancing algorithms a chance to work. With pretty much any other protocol (except peer-to-peer clients, which also mostly do client-side load balancing) there's a big temptation to have a single huge cloud that appears in as many places as possible. -Bill
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Richard A Steenbergen wrote: > The price being charged for the public exchange ports is non-trivial Only at the (very few) commercial exchanges. The vast majority are free or of trivial expense. But some people really like to lose money, since then they get to hang out with VCs and feel like movers and shakers, rather than feeling like peons who have to actually turn a profit. > Personally I've never understood why US exchange port operators havn't > insisted on some kind of "80% utilization over Xth percentile and you must > upgrade" rule. No idea. It works well elsewhere. I think people here just don't like the idea of being told what to do. -Bill
Re: ultradns reachability
In a message written on Fri, Jul 02, 2004 at 05:55:13PM -0700, Matt Ghali wrote: > DNS traffic, surprisingly, is not very "fat". It is no HTTP nor SMTP. > > The engineering behind appropriately sizing a unicast fallback would > be pretty trivial, especially compared to building a somewhat-robust > anycast architecture. This statement may be true for many DNS servers, but I suspect it is completely false for the roots, or for the GTLD's. Perhaps the folks from .org or from f-root would like to comment on how hard it would be to handle the whole load from a single box, particularly when you consider they are all high profile DDoS targets as well. If it were trivial, more GTLD's would be doing it. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org pgpIUv2wbknuR.pgp Description: PGP signature
Re: concern over public peering points [WAS: Peering point speed publicly available?]
At 02:07 AM 7/3/2004 -0400, Richard A Steenbergen wrote: b) The price being charged for the public exchange ports is non-trivial (especially compared to the cost of transit these days!), and is billed on a port basis instead of a usage basis (at least in the US). Since public peering is treated as a "necessary evil", with traffic moved to much more economical private peers when they start getting full, no one wants to provision extra capacity ahead of demand (in fact, in the US it is exceedingly rare to see anyone with 2 ports on a single public exchange). As one of the folks who gets questioned by Sales all the time about the reasons behind the multiple shared fabric ports at the IXs I'll gladly explain why we have 14 in the US at present and are preparing for ~5-10 abroad. 1. Trials. There are some networks who are not ready to properly manage private peering, they should be but they are not. A 90-day 'try before you buy' helps reduce the nickel & diming to a budget that remote hands and inventory adjustments chew. IMHO, if they do not have their operations activities in order they should not be a peer and that is one of the criteria we verify. 2. PNI sizing. Some networks really don't know how much traffic they will have to other networks when adding peering relations. If they argue about sizing it is best to drop them on to shared fabrics first to confirm with visuals what is flowing. 3. PNIs do not guarantee congestion avoidance. Unfortunately private peering does not remove congestion with some networks, it just shifts it. The peering relations community is well networked with each other. We know which network offenders have capacity issues regardless of public or private options. 4. International peers. Rarely are two network foot prints or goals for business the same. I would rather make available the unique international routes to our customers than miss that opportunity by being a public peering snob. This also allows the view towards new markets which rely heavily on shared fabrics. While not customary in the US, many EU peering IXs are multiple interconnected buildings managed by a single IX vendor at the shared fabric layer. Connecting to the shared fabric is an easy way to reach those networks in various buildings without dark fiber complexities. 5. Costs. Private peering is expensive, don't let anyone fool you. There is a resource investment in human terms that is rarely calculated properly, all the way from planning of inventory to planning for capacity augments after the physical install. It is often difficult to capture the cost to roll all those fibers that are improperly installed. This I'm sure you are painfully aware of . 6. Management. Set a range of expectations on levels for monitoring, hardware, power, staff time, and capacity upgrade paths by designating some peers in a 'group' vs. monitoring all as individuals. I encourage authors of RFPs to stop placing such an unnecessary stigma on public peering. Those networks without the benefit of options for interconnecting should be penalized for failure to evolve. Quite likely they are not connected to the growing sources in the current peering game. What is this called... the bagel syndrome? -ren
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Richard A Steenbergen wrote: > b) The price being charged for the public exchange ports is non-trivial >(especially compared to the cost of transit these days!), and is billed >on a port basis instead of a usage basis (at least in the US). Since >public peering is treated as a "necessary evil", with traffic moved to >much more economical private peers when they start getting full, no one >wants to provision extra capacity ahead of demand (in fact, in the US >it is exceedingly rare to see anyone with 2 ports on a single public >exchange). This is counter intuitive to me altho perhaps I need to better understand the IX operators income model. If I were a colo company who also operated an IX I'd want to encourage people to use my IX and put as much traffic over it. The logic being that operators gravitate towards these high bandwidth exchange areas and that means new business. The encouragement here would be to make the IX cost quite small.. of course the other benefit of succeeding in getting a lot of operators and traffic on your IX is you can publicise the data to show why you're better (or as good as) your competitors.. This doesnt affect their income from colo, support, cross connects so why not do it? Steve
Re: Who broke .org?
On Jul 2, 2:48pm, Jeff Wasilko <[EMAIL PROTECTED]> wrote: > On Fri, Jul 02, 2004 at 02:38:12PM -0400, Patrick W Gilmore wrote: > > run .org, I just think a blanket statement "anycast is bad" is, well, > > bad.) > > I'd be totally happy to see a combination, too. It's just pretty > obvious that the current solution isn't reliable over the long-haul. At least the previous outage (a couple of weeks ago) had nothing to do with anycast, I was getting NXDOMAIN replies back, and no kind of fallback or non-anycast deployment would have helped. Anycast functionality is well understood, and is less likely to cause problems than the fact that the servers seem to be under a single "one typo breaks all" management system. True redundancy requires the two sets of servers to be managed by different people doing different things in different ways at different times; this too is well understood, and I'm a little dismayed this (or at least some weaker form thereof) seems not to be the case. (Note I'm saying "seem", I'm just guessing based on the observations I and other people have made.) I think a statement from UltraDNS and/or Public Interest Registry as to what caused the problems would have been, and still is, appropriate, considering the public interest aspect. -- Per
