RE: concern over public peering points [WAS: Peering point speed publicly available?]
A minitel - in the United States! Scott C. McGrath On Thu, 8 Jul 2004, Ian Dickinson wrote: Which almost begs the question - what's the oddest WTF?? anybody's willing to admit finding under a raised floor, or up in a ceiling or cable chase or similar location? (Feel free to change names to protect the guilty if need be:) Water -- about 8 of it... Air -- about 8 feet of it... In a comms room in a tunnel under London. Luckily for those working there, there was a ladder stored there too. The term 'raised floor' was never so apt. -- Ian Dickinson Development Engineer PIPEX [EMAIL PROTECTED] http://www.pipex.net
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Tue, 06 Jul 2004 08:46:49 EDT, Leo Bicknell [EMAIL PROTECTED] said: Everyone running their cable wherever they want with no controls, and abandoning it all in place makes a huge mess, and is one way to think about it. While clearing out the space that eventually ended up being repurposed for a supercomputer, we encountered a small run of Ethernet Classic - the thickwire stuff. We never did figure out how or why it got there (I doubt that anybody stashed it down there just for storage stretched straight out, with 3 vampire taps still attached), as the location in question was still cow pasture when we decided that all new cable would be thinwire (and we certainly had plenty of THAT under the floor, buried under all the cat-5...) And we're a small enough shop with low enough personnel turnover that rounding up *all* the possible co-conspirators and getting somebody to admit Ahh... now there's a story attached to that wire... usually doesn't take more than 3 or 4 pitchers of Guinness... ;) Which almost begs the question - what's the oddest WTF?? anybody's willing to admit finding under a raised floor, or up in a ceiling or cable chase or similar location? (Feel free to change names to protect the guilty if need be:) pgpWDspZXFvep.pgp Description: PGP signature
Re: concern over public peering points [WAS: Peering point speed publicly available?]
In message [EMAIL PROTECTED], Valdis.Kletni [EMAIL PROTECTED] writes: Which almost begs the question - what's the oddest WTF?? anybody's willing to admit finding under a raised floor, or up in a ceiling or cable chase or similar location? (Feel free to change names to protect the guilty if need be:) Water -- about 8 of it... We had a two-level area below the raised floor in the computer room. The deeper area was flooded; fortunately, there was only solid insulated cables in that section. If the water had reached the shallower area, where there were outlets, connectors, etc., it would have been a different story. --Steve Bellovin, http://www.research.att.com/~smb
RE: concern over public peering points [WAS: Peering point speed publicly available?]
Which almost begs the question - what's the oddest WTF?? anybody's willing to admit finding under a raised floor, or up in a ceiling or cable chase or similar location? (Feel free to change names to protect the guilty if need be:) Water -- about 8 of it... Air -- about 8 feet of it... In a comms room in a tunnel under London. Luckily for those working there, there was a ladder stored there too. The term 'raised floor' was never so apt. -- Ian Dickinson Development Engineer PIPEX [EMAIL PROTECTED] http://www.pipex.net
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Wednesday 07 July 2004 02:43 am, [EMAIL PROTECTED] wrote: Which almost begs the question - what's the oddest WTF?? anybody's willing to admit finding under a raised floor, or up in a ceiling or cable chase or similar location? (Feel free to change names to protect the guilty if need be:) Not really a WTF. At my last job while working at an earthstation in Texas where I had some equipment, I looked up from the raised floor and found myself staring at a scorpion. Being that I am from the Northeast where we don't seem to have those things, it pretty much scared the heck out of me. Gave the techs at the station a good laugh. -Patrick -- Patrick Muldoon Network/Software Engineer INOC (http://www.inoc.net) PGPKEY (http://www.inoc.net/~doon) Key ID: 0x370D752C Select * from users where clue 0 O Rows Returned
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Thu, 8 Jul 2004, Steven M. Bellovin wrote: Water -- about 8 of it... We had a two-level area below the raised floor in the computer room. The deeper area was flooded; fortunately, there was only solid snakes in the water, which had swum (swam?) in through the entrance facility for the building electric... 'fun'!
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Thu, 8 Jul 2004, Patrick Muldoon wrote: At my last job while working at an earthstation in Texas where I had some equipment, I looked up from the raised floor and found myself staring at a scorpion. Being that I am from the Northeast where we don't seem to have those things, it pretty much scared the heck out of me. Gave the techs at the station a good laugh. Sounds like they need to make cowboy boots standard attire down there :)
RE: concern over public peering points [WAS: Peering point speed publicly available?]
On Wednesday 07 July 2004 02:43 am, [EMAIL PROTECTED] wrote: Which almost begs the question - what's the oddest WTF?? anybody's willing to admit finding under a raised floor, or up in a ceiling or cable chase or similar location? (Feel free to change names to protect the guilty if need be:) Raccoons. Came in late one night and heard noises that I didn't really expect. Turns out the facility had diverse entrances and multiple conduits - and one of them had been exposed outside due to some erosion and had been damaged. We found little surprises for quite awhile after that. Undergarments and shoes. His and hers, but no other clothing. A crutch. Just one. On the not under the floor I was at a facility that had an enormous amount of open floor space and far too much air. (Wishful thinking) The ops staff moved all the grated tiles to a central area and used to play adult-sized air hockey complete with a rubber puck and sticks... but only late at night. -John
RE: concern over public peering points [WAS: Peering point speed publicly available?]
On Fri, 9 Jul 2004, John Ferriby wrote: The ops staff moved all the grated tiles to a central area and used to play adult-sized air hockey complete with a rubber puck and sticks... but only late at night. 'login;' ran a story about 4-5 years ago about some machine room in the UK (I think), something about playing cricket friday evenings... until somone hit one out of the 'park' tripping the emergency power off button for the machine room :(
Re: concern over public peering points [WAS: Peering point speed publicly available?]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2004-07-03, at 18.10, Mikael Abrahamsson wrote: It's when the exchange is being run by a separate entity that needs a marketing department, a well-paid staff of managers, technicians etc that price really goes up. All this to basically manage a simple ethernet switch that needs some patching a couple of times a month at maximum. For quite some months I have spent time thinking on this particular issue. And one thing have struck me with the discussions of staffing levels. It is always true that if your day job get's payed by some other revenue generating business, running a IX with that staff should be easy. That is cross-subsidation and there is no need for recovering costs for the IXP. At the same time, there are a number of roles you can only take that far in that way. One of the most obvious ones is growing the membership number. Now, it's not always the case that an increased membership number benefits the members, but I am willing to claim that it is in most cases. Reason is simply that the cost of running the exchange is not directly proportional to the number of members. So more members means less cost per member for a non-for-profit IX. Also, more members should increase the value for the other members as they have the possibility to peer-away more traffic. Now, I am willing to claim that you can only get new members by reputation up to a certain point. After that you will need to start to actively go out and find them, and deal with them. This will cost you money. I have with great interest followed how non-for-profit IXPs in Europe have started employing marketing staff. I have no idea if this pays off for them, but I suspect it does. - - kurtis - -BEGIN PGP SIGNATURE- Version: PGP 8.0.3 iQA/AwUBQOpDiaarNKXTPFCVEQJDGwCfXqZw3+7YFuDPNiuvUONfVYi+mYkAnj6h Ud8VxItMH8qNXqrObTY6inSK =pr29 -END PGP SIGNATURE-
Re: concern over public peering points [WAS: Peering point speed publicly available?]
I'm wondering why you think that the fiber over the ceiling tile is somehow less tracked, maintained, monitored, documented, etc., than any other fiber in the network? If someone was really concerned about trackability, etc., then I suspect they would invent a number for that cable, put a record in their circuit database, and use their nifty label maker machine to put labels every meter along the cable's length stating the circuit number, and NOC contact info. All of that work is still not much more than the effort of stringing the cable but it makes the whole architecture a lot more scalable. So there is a middle ground between flinging cables around and paying $1000 per month for a cross-connect... Middle grounds are nice places to play in. Lot's of variety, lot's of possibilities. --Michael Dillon
Re: concern over public peering points [WAS: Peering point speed publicly available?]
In any case, I am going to pull a randy here and strongly encourage my competitors to deploy this ethernet over ceiling tile engineering methodology. Funny thing is, that there are lots of competitors doing what randy strongly encourages them to do and they stay in business. I think it's all a question of scale. If you are at the top end of the scale or if you seriously intend to get to the top end of the scale, then it's good to be anal about these things and strive for the absolute best practice and most scalable engineering. On the other hand there is limited room at the top and most people are happy to run a business on a somewhat smaller scale. One size does not fit all. --Michael Dillon
Re: concern over public peering points [WAS: Peering point speed publicly available?]
In a message written on Tue, Jul 06, 2004 at 04:32:14AM +, vijay gill wrote: Paul, I think you took a left at the pass and went down the wrong road here. I am not saying ethernet doesn't scale or even vni/pni doesn't scale, but the mentality embodied in the approach throw it over the wall doesn't bode well if you are to scale. Throw it over the wall can be interpreted many ways. Everyone running their cable wherever they want with no controls, and abandoning it all in place makes a huge mess, and is one way to think about it. However, there are lots of telco MMR's, with either rows of racks or cages where every party runs their own fiber. Typically trays are provided in the colo cost, and the parties run the fiber in the trays and use the fiber management, label their jumpers, and more often than not pull out unused cables. If cages are involved dropping the cable over the cage is a common practice. Walking into these facilities you find they are generally neat and organized. I believe the problem Vijay is referencing isn't throw it over the wall, but rather where people have to hide the fact that they are throwing it over the wall. When some colo providers want to do things like charge a 0-mile local loop for a fiber across the room people think it's too much, and run their own over the wall fiber. However since it's technically not allowed it's hidden, unlabeled, abandoned when unused, and creates a huge mess. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org pgp5FS9twMz0r.pgp Description: PGP signature
Re: concern over public peering points [WAS: Peering point speed publicly available?]
--On Tuesday, July 06, 2004 08:46 -0400 Leo Bicknell [EMAIL PROTECTED] wrote: Everyone running their cable wherever they want with no controls, and abandoning it all in place makes a huge mess, and is one way to think about it. [snipped] I believe the problem Vijay is referencing isn't throw it over the wall, but rather where people have to hide the fact that they are throwing it over the wall. When some colo providers want to do things like charge a 0-mile local loop for a fiber across the room people think it's too much, and run their own over the wall fiber. However since it's technically not allowed it's hidden, unlabeled, abandoned when unused, and creates a huge mess. Thanks. Precisely the issue. Being humans involved in this, there is a tendency to sometimes hack around a problem and then leave it in place. I know I am susceptible to this and have to be on guard against this mentality at all times. And I've seen plenty of this in various orgs. The key here is to maintain an engineering discipline and be on constant guard against 'just this once' kind of thought. There should be no negotiations with yourself. Even the best of intentions lead to massive entropy when doing hacks around issues. Temporary fixes aren't. /vijay
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Tue, 6 Jul 2004, vijay gill wrote: Temporary fixes aren't. so true.. hands up anyone who doesnt have something in their network/systems labelled as 'legacy' ;) Steve
RE: concern over public peering points [WAS: Peering point speed publicly available?]
Thanks. Precisely the issue. Being humans involved in this, there is a tendency to sometimes hack around a problem and then leave it in place. I know I am susceptible to this and have to be on guard against this mentality at all times. And I've seen plenty of this in various orgs. The key here is to maintain an engineering discipline and be on constant guard against 'just this once' kind of thought. There should be no negotiations with yourself. Even the best of intentions lead to massive entropy when doing hacks around issues. Temporary fixes aren't. /vijay Setting aside the issue of abandoning media after you stop using it, a cable run based on a handshake between two tenants in a telco hotel CAN lead to nightmares when it goes down. On the other hand, if you figure out a way to document it, and have field support lined up, it may turn out to be more easily restored than an official interconnect. :-)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
Careful with those invokations, Vijay. As we have seen before in previous lives, and I'm pretty sure stephen stuart will step in, normalizing throw the ethernet over the wall school of design just leads to an incredible amount of pain when trying to operate, run and actually document what you've got. The various replies have largely covered what I would say; that it's all about the OAM. Yes, in the previous life that you mention (known currently as the good old days), some pain was suffered. If there is a spectrum roughly described as: - have no standards or documentation (and none of the pain of developing, following, maintaining them) and spend all your time doing discovery each time you have a problem to solve (so that all of your pain results from having no operational consistency) - have standards that you don't follow and documentation that you don't maintain and constantly trip over exception cases (suffer pain on both ends of the spectrum) - have standards that are followed and documentation that is maintained and achieve a high level of operational consistency (this is widely regarded as better) then the pain that you describe came from moving along that spectrum. The pain came from moving, not necessarily from the direction that we chose. We moved in the direction that we did because the goals that we set for ourselves demanded it. Hopefully the folks still there continue to reap the benefits of that work. Each organization chooses (whether consciously or not) a point in the spectrum described above and operates there. They compete in the marketplace without that choice being a significant differentiator; an organization that lacks design skills might compensate by being able to debug problems quickly, for example, such that externally measurable metrics that drive purchasing decisions are roughly equivalent. There is no One Truth; you try to make your organizational strengths work for you to maximum benefit, while not getting tripped up by your weaknesses. What *is* a differentiator is how well you execute at the point in the spectrum that you've chosen. That choice is made over and over again within the lifecycle of an organization. The wheel turns, and administrations come and go, moving one way or the other, using the previous administration did X and we must do Y as justification for desired resources to travel in either direction (or to stay in one place, with the appropriate label engineering to make it look as though motion will occur). All the while, the benefits and drawbacks of various aspects of various choices will be debated on the NANOG mailing list. There's an analogy to samsara hiding in there, for those that like analogies. I'd elaborate, but it's time to take the dogma for a walk. Stephen
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sun, 4 Jul 2004, Bill Woodcock wrote: Go back and think about the purpose of an exchange: it's an economic optimization over transit. It's the value-add that lets someone who buys transit sell a service that's of greater value yet lesser cost than what they buy. Now, what's an exchange that costs more money? Less effective. So charging more money is just a way to make an exchange less effective, not more effective. Some people who don't bother to run the numbers think that reliability is a justification for charging money at an exchange. I'd encourage them to run the numbers. An exchange which costs twice as much money needs to be twice as reliable to be equally effective. Ask yourself how many exchanges there are that drop more than half the bits on the floor, before you think of charging twice as much money. In other words, if it's not broke, don't fix it. I'm going to quibble with Bill a bit on this. I'm going to attempt to do so carefully, both because I work for him, and because this is an area he knows much better than I do. ;) Peering is often said to help a network (and thus add value) in two ways, by increasing performance and by reducing costs. Performance is said to increase because other networks are brought closer. Costs are said to go down because a network starts getting connectivity for free that it would otherwise be paying for. In well connected urban areas of the US, for all but the biggest networks, it's no longer clear that either of these arguments are valid. The performance arguments are probably more controversial. The arguments are that shortening the path between two networks increases performance, and that removing an extra network in the middle increases reliability. The first argument holds relatively little water, since it's in many cases only the AS Path (not really relevant for packet forwarding performance) that gets shortened, rather than the number of routers or even the number of fiber miles. If traffic goes from network A, to network A's router at an exchange point, to network C, that shouldn't be different performance-wise from the traffic going from network A, to Network B's router at the exchange point, to Network C. Assuming none of the three networks are underprovisioning, the ownership of the router in the middle shouldn't make much difference. The reliability argument is probably more valid -- one less network means one less set of engineers to screw something up, but the big transit networks tend to be pretty reliable these days, and buying transit from two of them should be quite safe. The pricing issues are simpler. There's a cost to transit (which is, to some degree, paying some other network to do your peering for you), and there's a cost to peering. Without a clear qualitative difference between the two, peering needs to be cheaper to make much sense. The costs of transit involve not just what gets paid to the transit provider for the IP transit, but also the circuit to the transit provider, the router interface connecting to the transit provider, engineering time to maintain the connection and deal with the transit provider if they have issues, and so forth. Costs of peering include not just the cost of the exchange port, but also the circuit to get to the exchange switch, sometimes colo in the exchange facility, engineering time to deal with the connection and deal with the switch operator if there are issues, and time spent dealing with each individual peer, both in convincing them to turn the session up, and dealing with problems affecting the session. Even if the port on the exchange switch were free, there would be some scenarios in which peering would not be cheaper than transit. The situation changes considerably in less developed areas. The transit costs tend to be a lot higher (largely due to increased long-haul circuit costs), and there's a significant performance cost to having your traffic go hundreds or thousands of miles to get across town. The argument against free exchanges is, I think, more of an argument in favor of full-service facilities, and the savings they provide in terms of operational engineering time. If there's a problem at 3 am at PAIX, Equinix, or NOTA (to pick three well-known North American commercial exchange operators), it's easy to pick up the phone and get it resolved. When dealing with volunteers, or with an organization that doesn't have the budget for a 24/7 paid staff, there's at least a perception that it may be hard to find somebody who will make fixing somebody else's problem their top priority. Again, it becomes a matter of plugging that cost into the cost comparison, and figuring out whether it costs more to peer with or without that level of service. I would bet that there is more than enough business available to cover the costs of intelligent spending. While I won't categorically dub that an oxymoron, I'd say that the possibility of their
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 09:24:17PM +0200, Mikael Abrahamsson wrote: On Sat, 3 Jul 2004, Stephen J. Wilcox wrote: [snip] IXes are not for top carriers ^^^ Like the economy, perhaps this is different in .se. But this is NAnog to which you are sending the message, and the above statement is incorrect. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On 7/5/04 1:18 AM, Steve Gibbard [EMAIL PROTECTED] wrote: The performance arguments are probably more controversial. The arguments are that shortening the path between two networks increases performance, and that removing an extra network in the middle increases reliability. The first argument holds relatively little water, since it's in many cases only the AS Path (not really relevant for packet forwarding performance) that gets shortened, rather than the number of routers or even the number of fiber miles. If traffic goes from network A, to network A's router at an exchange point, to network C, that shouldn't be different performance-wise from the traffic going from network A, to Network B's router at the exchange point, to Network C. Assuming none of the three networks are underprovisioning, the ownership of the router in the middle shouldn't make much difference. The reliability argument is probably more valid -- one less network means one less set of engineers to screw something up, but the big transit networks tend to be pretty reliable these days, and buying transit from two of them should be quite safe. I believe that peering does lead to a more robust network and somewhat better performance. Being heavily peered means that when one of my transit providers suffers a network 'event', I am less affected. Also, just because I'm sitting at a network exchange point (and take my transit there) doesn't mean that's where my transit networks peer. Quite often, I see traffic going to Stockton or Sacramento through one of my transit connections to be delivered to a router just a few cages away at PAIX. The pricing issues are simpler. There's a cost to transit (which is, to some degree, paying some other network to do your peering for you), and there's a cost to peering. Without a clear qualitative difference between the two, peering needs to be cheaper to make much sense. The costs of transit involve not just what gets paid to the transit provider for the IP transit, but also the circuit to the transit provider, the router interface connecting to the transit provider, engineering time to maintain the connection and deal with the transit provider if they have issues, and so forth. Costs of peering include not just the cost of the exchange port, but also the circuit to get to the exchange switch, sometimes colo in the exchange facility, engineering time to deal with the connection and deal with the switch operator if there are issues, and time spent dealing with each individual peer, both in convincing them to turn the session up, and dealing with problems affecting the session. Even if the port on the exchange switch were free, there would be some scenarios in which peering would not be cheaper than transit. When we established our connection at PAIX, peering bandwidth was a factor of 20 cheaper than transit. Now they're at parity. Unfortunately, some *IX operators haven't seen fit to become more competitive on pricing to keep peering more economical than average transit pricing. $5000 for an ethernet switch port? It makes me long for the days of throwing ethernet cables over the ceiling to informally peer with other networks in a building. In the 'bad' old days of public exchanges (even the ad hoc ones), most of the problems were with the design and traffic capacity of the equipment itself (not a real problem now), not with actual 'operations'.
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Mon, Jul 05, 2004 at 10:55:42AM -0700, joe mcguckin wrote: $5000 for an ethernet switch port? It makes me long for the days of throwing ethernet cables over the ceiling to informally peer with other networks in a Throwing ethernet cables over the ceiling does not scale. /vijay
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Jul 5, 2004, at 2:02 PM, vijay gill wrote: On Mon, Jul 05, 2004 at 10:55:42AM -0700, joe mcguckin wrote: $5000 for an ethernet switch port? It makes me long for the days of throwing ethernet cables over the ceiling to informally peer with other networks in a Throwing ethernet cables over the ceiling does not scale. Sure it does. The question is: How far does it scale? Nothing scales to infinity, and very, very few things do not scale past the degenerate case of 1. If you s/ethernet cables/optical fibers/, it scales even further. Especially since this is not being used for all his traffic. Not everyone needs a terabit of exit capacity. Guaranteeing everything you do is close to infinitely scalable is a Bad Idea for people who do not. And even if you do need a terabit of exit capacity, nothing wrong with the occasional OC768 routed through the ceiling. :) -- TTFN, patrick
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Jul 5, 2004, at 5:00 PM, Patrick W Gilmore wrote: On Jul 5, 2004, at 2:02 PM, vijay gill wrote: Throwing ethernet cables over the ceiling does not scale. Sure it does. The question is: How far does it scale? Nothing scales to infinity, and very, very few things do not scale past the degenerate case of 1. You need to take into account all of the aspects of the complexity that you introduce when you throw that fiber over the wall tho. While the fiber installation is simple enough, you have now created other problems: who will maintain it? Who knows it is there? Who knows that it is there in the other organization? Who needs to know about it within your own organization? How is tracked? Who does the NOC call when it goes bad? While it may be a single exception to your network architecture, if it is an exception that 100 people need to know about, then I'd argue that it doesn't scale. The fun and games that we had in Ye Olden Days o' the Internet simply are not workable when you are coordinating with hundreds of other employees. Put another way, scalability can never overlook the human element. Tony
Re: concern over public peering points [WAS: Peering point speed publicly available?]
[EMAIL PROTECTED] (vijay gill) writes: Throwing ethernet cables over the ceiling does not scale. i think it's important to distinguish between things aol and uunet don't think are good for aol and uunet and things that aren't good for anybody. what i found through my PAIX experience is that the second tier is really quite deep and broad, and that the first tier doesn't ignore them like their spokesmodels claim they do. what i found in helping to hone the ssni approach is that while public peering ethernet style is dead, vni/pni peering is alive and well. anyone who does not agree is free to behave that way. but it's not useful to try to dissuade cooperating adults from peering any way they want to. the interesting evolutionary aspect to this is that vni/pni peering starting with atm and moving to pni doesn't work at all, because atm by and large has a high cost per bit at the interface, and a low top end, and usually doesn't mandate co-location. but vni/pni peering over 802.1Q usually does succeed, because of the low cost per bit at the interface, the obscenely high top end, and the greater likelihood that the vni parties are co-located and so can switch to pni when the traffic volume warrants it. i've been told that if i ran a tier-1 i would lose my love for the vni/pni approach, which i think scales quite nicely even when it involves an ethernet cable through the occasional ceiling. perhaps i'll eat these words when and if that promotion comes through. meanwhile, disintermediation is still my favorite word in the internet dictionary. i like it when one's competitors are free to do business with each other, it leads to more and better innovation. -- Paul Vixie
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Jul 5, 2004, at 8:35 PM, Tony Li wrote: On Jul 5, 2004, at 5:00 PM, Patrick W Gilmore wrote: On Jul 5, 2004, at 2:02 PM, vijay gill wrote: Throwing ethernet cables over the ceiling does not scale. Sure it does. The question is: How far does it scale? Nothing scales to infinity, and very, very few things do not scale past the degenerate case of 1. You need to take into account all of the aspects of the complexity that you introduce when you throw that fiber over the wall tho. While the fiber installation is simple enough, you have now created other problems: who will maintain it? Who knows it is there? Who knows that it is there in the other organization? Who needs to know about it within your own organization? How is tracked? Who does the NOC call when it goes bad? While it may be a single exception to your network architecture, if it is an exception that 100 people need to know about, then I'd argue that it doesn't scale. The fun and games that we had in Ye Olden Days o' the Internet simply are not workable when you are coordinating with hundreds of other employees. Put another way, scalability can never overlook the human element. I'm wondering why you think that the fiber over the ceiling tile is somehow less tracked, maintained, monitored, documented, etc., than any other fiber in the network? Put another way: Just because I am not paying someone 1000s of $$ a month to watch it for me does not mean the human element is ignored. In fact, I have seen many cases where people lost track of interconnects where they were paying lots of money for someone else to watch them. So maybe the strange ones are better =) If you do not want to throw cables over the ceiling in your network, then by all means do not. I have repeated many times here and elsewhere: Your network, your decision. And for those of us who can track maintain zero-dollar interconnects, please do not begrudge us the cost savings. -- TTFN, patrick
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Tue, Jul 06, 2004 at 01:43:14AM +, Paul Vixie wrote: [EMAIL PROTECTED] (vijay gill) writes: Throwing ethernet cables over the ceiling does not scale. i think it's important to distinguish between things aol and uunet don't think are good for aol and uunet and things that aren't good for anybody. what i found through my PAIX experience is that the second tier is really quite deep and broad, and that the first tier doesn't ignore them like their spokesmodels claim they do. Paul, I think you took a left at the pass and went down the wrong road here. I am not saying ethernet doesn't scale or even vni/pni doesn't scale, but the mentality embodied in the approach throw it over the wall doesn't bode well if you are to scale. not agree is free to behave that way. but it's not useful to try to dissuade cooperating adults from peering any way they want to. i've been told that if i ran a tier-1 i would lose my love for the vni/pni approach, which i think scales quite nicely even when it involves an ethernet cable through the occasional ceiling. perhaps i'll eat these words when and if that promotion comes through. meanwhile, disintermediation is still my favorite word in the internet dictionary. i like it when one's competitors As we have seen before in previous lives, and I'm pretty sure stephen stuart will step in, normalizing throw the ethernet over the wall school of design just leads to an incredible amount of pain when trying to operate, run and actually document what you've got. I thought it was illustrative to take a look at some of the other messages in this thread. People rushing in to argue the scale comment when the actual heart of the matter was something else entirely, which apparently only Tony managed to get. In any case, I am going to pull a randy here and strongly encourage my competitors to deploy this ethernet over ceiling tile engineering methodology. /vijay
Re: concern over public peering points [WAS: Peering point speed publicly available?]
Kind of summarizes why we are still heavy on the best effort side of the equation. -M Regards, -- Martin Hannigan (c) 617-388-2663 VeriSign, Inc. (w) 703-948-7018 http://www.verisign.com/ -Original Message- From: [EMAIL PROTECTED] [EMAIL PROTECTED] To: Paul Vixie [EMAIL PROTECTED] CC: [EMAIL PROTECTED] [EMAIL PROTECTED] Sent: Mon Jul 05 21:32:14 2004 Subject: Re: concern over public peering points [WAS: Peering point speed publicly available?] On Tue, Jul 06, 2004 at 01:43:14AM +, Paul Vixie wrote: [EMAIL PROTECTED] (vijay gill) writes: Throwing ethernet cables over the ceiling does not scale. i think it's important to distinguish between things aol and uunet don't think are good for aol and uunet and things that aren't good for anybody. what i found through my PAIX experience is that the second tier is really quite deep and broad, and that the first tier doesn't ignore them like their spokesmodels claim they do. Paul, I think you took a left at the pass and went down the wrong road here. I am not saying ethernet doesn't scale or even vni/pni doesn't scale, but the mentality embodied in the approach throw it over the wall doesn't bode well if you are to scale. not agree is free to behave that way. but it's not useful to try to dissuade cooperating adults from peering any way they want to. i've been told that if i ran a tier-1 i would lose my love for the vni/pni approach, which i think scales quite nicely even when it involves an ethernet cable through the occasional ceiling. perhaps i'll eat these words when and if that promotion comes through. meanwhile, disintermediation is still my favorite word in the internet dictionary. i like it when one's competitors As we have seen before in previous lives, and I'm pretty sure stephen stuart will step in, normalizing throw the ethernet over the wall school of design just leads to an incredible amount of pain when trying to operate, run and actually document what you've got. I thought it was illustrative to take a look at some of the other messages in this thread. People rushing in to argue the scale comment when the actual heart of the matter was something else entirely, which apparently only Tony managed to get. In any case, I am going to pull a randy here and strongly encourage my competitors to deploy this ethernet over ceiling tile engineering methodology. /vijay
Re: concern over public peering points [WAS: Peering point speed publicly available?]
i've been told that if i ran a tier-1 i would lose my love for the vni/pni approach, which i think scales quite nicely even when it involves an ethernet cable through the occasional ceiling. perhaps i'll eat these words when and if that promotion comes through. meanwhile, disintermediation is still my favorite word in the internet dictionary. i like it when one's competitors As we have seen before in previous lives, and I'm pretty sure stephen stuart will step in, normalizing throw the ethernet over the wall school of design just leads to an incredible amount of pain when trying to operate, run and actually document what you've got. that only happens if you grow, though. in other words it's a nice problem to have. i guess i'll defer to the people who have started from scratch and gotten all the way up to having the problem you describe. i met a lot of them though my PAIX experience. you should defer to them on this point also. I thought it was illustrative to take a look at some of the other messages in this thread. People rushing in to argue the scale comment when the actual heart of the matter was something else entirely, which apparently only Tony managed to get. i understood and agreed with tony's point. and during my mfn experience i was quite supportive of normalization, and was quite damning in my curses of what had been doing done in the early/growth period before my time. but that doesn't mean that the occasional ethernet cable over the ceiling isn't the right business decision for people who have done what many here have not done, which is to say, started from scratch. In any case, I am going to pull a randy here and strongly encourage my competitors to deploy this ethernet over ceiling tile engineering methodology. i would think that was puerile and shallow no matter who actually said it.
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 01:00:35AM -0400, Patrick W Gilmore wrote: On Jul 2, 2004, at 9:31 PM, Stewart, William C (Bill), RTSLS wrote: Also, if you're dealing with ISPs that use public peering points, those may be a performance concern, but in the US that's mostly not Tier1-Tier1. (Linx is a different case entirely, assuming you want your traffic to be in London.) Any particular reason you would worry about public peering points these days? The FDDI MAEs are dead, there is no head of line blocking any more. Every ethernet or ATM switch running a NAP I've seen in the last ... nearly half a decade is more than capable of passing all bits going through it without a problem, and then some. What is with people in this industry, who latch onto an idea and won't let go? If someone was talking about 80286 based machines in 2004 we would all be in utter disbelief, but you can still routinely find people talking about the MAEs and congested NAPs. There might be a concern that, for instance, a provider would show up to a NAP, connect at GigE, then peer with 2 gigabits of traffic. But I fail to see why that is the public fabric's fault, or why things would be any different on private peering. The provider knows when their connection is congested, be it an ethernet to a NAP or an OC to another router. I also have not seen that affect the packets not going to the congested port (unlike some older NAPs). a) Exchange points make a living convincing people to buy their product just like everyone else. When stupid people who don't know what they're doing buy transit, no one cares. When these same people who really don't know how to peer or manage their capacity start jumping on the save money or improve performance bandwagon without finding someone experienced to run it, they do stupid things. :) b) The price being charged for the public exchange ports is non-trivial (especially compared to the cost of transit these days!), and is billed on a port basis instead of a usage basis (at least in the US). Since public peering is treated as a necessary evil, with traffic moved to much more economical private peers when they start getting full, no one wants to provision extra capacity ahead of demand (in fact, in the US it is exceedingly rare to see anyone with 2 ports on a single public exchange). Personally I've never understood why US exchange port operators havn't insisted on some kind of 80% utilization over Xth percentile and you must upgrade rule. Since you don't normally have an idea how hot your peer is running their public port, you're really putting a *lot* of faith in your peers' ability to manage their traffic when you peer with them over a public exchange. Given how poorly some folks do this, and how quickly a congested port can degrate the reputation of an exchange point, it seems like this would at least be a very basic safety net (doesn't help if they only have 1 OC12 of backhaul off of that GigE port, but still better than nothing). Plus as I'm sure we all know the price of the exchange point switch port is covered by the first months' fees. What we're really paying for is the faith that the EP operator will keep things up and running, prevent forwarding loops, check for bad things being broadcasted, maybe invest in a bigger switch down the road, and be able to convince others to join so that there is a reason to bother peering there, etc. The extra cost of the ports is really quite trivial. Public NAPs got a bad name many years ago because a few of them were poorly run, and some other ones had some technical difficulties, and some providers intentionally congested their public ports so they could say see, public peering sucks, and lots of other reasons. Some still do. At the very least, I can personally think of at least 4 different folks with public GigE exchange ports sitting at 920-960Mbps peak *RIGHT NOW*. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
PWG Date: Sat, 3 Jul 2004 01:00:35 -0400 PWG From: Patrick W Gilmore PWG Any particular reason you would worry about public peering PWG points these days? ANES, perhaps? Those who finally found old NANOG-L and i-a archives have decided public peering is bad. H let's see cheap, uncongested public peering -vs- expensive private peering. Assuming fixed amount of money to spend, which buys more? There. Now we just need to wait a few more years for the public peering is good mentality to spread. Hopefully that will still be the case at that time. :-) PWG There might be a concern that, for instance, a provider PWG would show up to a NAP, connect at GigE, then peer with 2 PWG gigabits of traffic. But I fail to see why that is the PWG public fabric's fault, or why things would be any different PWG on private peering. The provider knows when their *nods* Private would be worse. Even collocation + overpriced $500/mo fiber x-c compares favorably with metro OC3. You've gotta admit, though: It's funny watching someone proclaim we avoid public peering! when their $149/mo dedicated server lives in a PAIX suite, unbeknowst to them. :-) I guess uncongested public peering technically _is_ avoiding congested public peering... Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: concern over public peering points [WAS: Peering point speed publicly available?]
RAS Date: Sat, 3 Jul 2004 02:07:06 -0400 RAS From: Richard A Steenbergen RAS What is with people in this industry, who latch onto an idea RAS and won't let go? If someone was talking about 80286 based RAS machines in 2004 we would all be in utter disbelief, but you RAS can still routinely find people talking about the MAEs and RAS congested NAPs. Can I get a class C with that? [ snip ] RAS Given how poorly some folks do this, and how quickly a RAS congested port can degrate the reputation of an exchange RAS point, it seems like this would at least be a very basic RAS safety net (doesn't help if they only have 1 OC12 of RAS backhaul off of that GigE port, but still better than RAS nothing). To think some of us thought exchanges would save providers from tyrannical ILEC loops. ;-) Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _ DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked.
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Richard A Steenbergen wrote: b) The price being charged for the public exchange ports is non-trivial (especially compared to the cost of transit these days!), and is billed on a port basis instead of a usage basis (at least in the US). Since public peering is treated as a necessary evil, with traffic moved to much more economical private peers when they start getting full, no one wants to provision extra capacity ahead of demand (in fact, in the US it is exceedingly rare to see anyone with 2 ports on a single public exchange). This is counter intuitive to me altho perhaps I need to better understand the IX operators income model. If I were a colo company who also operated an IX I'd want to encourage people to use my IX and put as much traffic over it. The logic being that operators gravitate towards these high bandwidth exchange areas and that means new business. The encouragement here would be to make the IX cost quite small.. of course the other benefit of succeeding in getting a lot of operators and traffic on your IX is you can publicise the data to show why you're better (or as good as) your competitors.. This doesnt affect their income from colo, support, cross connects so why not do it? Steve
Re: concern over public peering points [WAS: Peering point speed publicly available?]
At 02:07 AM 7/3/2004 -0400, Richard A Steenbergen wrote: b) The price being charged for the public exchange ports is non-trivial (especially compared to the cost of transit these days!), and is billed on a port basis instead of a usage basis (at least in the US). Since public peering is treated as a necessary evil, with traffic moved to much more economical private peers when they start getting full, no one wants to provision extra capacity ahead of demand (in fact, in the US it is exceedingly rare to see anyone with 2 ports on a single public exchange). hi ras! As one of the folks who gets questioned by Sales all the time about the reasons behind the multiple shared fabric ports at the IXs I'll gladly explain why we have 14 in the US at present and are preparing for ~5-10 abroad. 1. Trials. There are some networks who are not ready to properly manage private peering, they should be but they are not. A 90-day 'try before you buy' helps reduce the nickel diming to a budget that remote hands and inventory adjustments chew. IMHO, if they do not have their operations activities in order they should not be a peer and that is one of the criteria we verify. 2. PNI sizing. Some networks really don't know how much traffic they will have to other networks when adding peering relations. If they argue about sizing it is best to drop them on to shared fabrics first to confirm with visuals what is flowing. 3. PNIs do not guarantee congestion avoidance. Unfortunately private peering does not remove congestion with some networks, it just shifts it. The peering relations community is well networked with each other. We know which network offenders have capacity issues regardless of public or private options. 4. International peers. Rarely are two network foot prints or goals for business the same. I would rather make available the unique international routes to our customers than miss that opportunity by being a public peering snob. This also allows the view towards new markets which rely heavily on shared fabrics. While not customary in the US, many EU peering IXs are multiple interconnected buildings managed by a single IX vendor at the shared fabric layer. Connecting to the shared fabric is an easy way to reach those networks in various buildings without dark fiber complexities. 5. Costs. Private peering is expensive, don't let anyone fool you. There is a resource investment in human terms that is rarely calculated properly, all the way from planning of inventory to planning for capacity augments after the physical install. It is often difficult to capture the cost to roll all those fibers that are improperly installed. This I'm sure you are painfully aware of G. 6. Management. Set a range of expectations on levels for monitoring, hardware, power, staff time, and capacity upgrade paths by designating some peers in a 'group' vs. monitoring all as individuals. I encourage authors of RFPs to stop placing such an unnecessary stigma on public peering. Those networks without the benefit of options for interconnecting should be penalized for failure to evolve. Quite likely they are not connected to the growing sources in the current peering game. What is this called... the bagel syndrome? -ren
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Richard A Steenbergen wrote: The price being charged for the public exchange ports is non-trivial Only at the (very few) commercial exchanges. The vast majority are free or of trivial expense. But some people really like to lose money, since then they get to hang out with VCs and feel like movers and shakers, rather than feeling like peons who have to actually turn a profit. Personally I've never understood why US exchange port operators havn't insisted on some kind of 80% utilization over Xth percentile and you must upgrade rule. No idea. It works well elsewhere. I think people here just don't like the idea of being told what to do. -Bill
Re: concern over public peering points [WAS: Peering point speed publicly available?]
Bill Woodcock writes on 7/3/2004 7:02 PM: On Sat, 3 Jul 2004, Richard A Steenbergen wrote: The price being charged for the public exchange ports is non-trivial Only at the (very few) commercial exchanges. The vast majority are free or of trivial expense. But some people really like to lose money, since then they get to hang out with VCs and feel like movers and shakers, rather than feeling like peons who have to actually turn a profit. Hah. http://www.nixi.org - described to a T. srs -- suresh ramasubramanian [EMAIL PROTECTED] gpg EDEDEFB9 manager, security and antispam operations, outblaze ltd
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 08:28:50AM -0400, ren wrote: At 02:07 AM 7/3/2004 -0400, Richard A Steenbergen wrote: b) The price being charged for the public exchange ports is non-trivial (especially compared to the cost of transit these days!), and is billed on a port basis instead of a usage basis (at least in the US). Since public peering is treated as a necessary evil, with traffic moved to much more economical private peers when they start getting full, no one wants to provision extra capacity ahead of demand (in fact, in the US it is exceedingly rare to see anyone with 2 ports on a single public exchange). hi ras! As one of the folks who gets questioned by Sales all the time about the reasons behind the multiple shared fabric ports at the IXs I'll gladly explain why we have 14 in the US at present and are preparing for ~5-10 abroad. You're definitely one of the rare few, especially given your size. In Europe it seems far more common for people to provision multiple ports and make certain they have capacity. In the US, even the couple of other folks I can think of who actually decided to provision multiple ports on the modern exchanges we're thinking of ended up sitting with congestion for some number of weeks before they actually did it. The general line of thinking here is ok exchange port is getting full, lets move someone big to a PNI. Are there even any exchange points in the US who are actually doing 10GE right now (major and production, not someone tinkering)? One way or another, there is definitely room for improvement in the technology of public peering. Then again, with some classic exchanges (that are still considered viable, aka not mae's, aads, pbnap, etc) still charging the same prices they were back in 1999, aka more than transit, perhaps there is room for improvement in the financial model as well. :) 5. Costs. Private peering is expensive, don't let anyone fool you. There is a resource investment in human terms that is rarely calculated properly, all the way from planning of inventory to planning for capacity augments after the physical install. It is often difficult to capture the cost to roll all those fibers that are improperly installed. This I'm sure you are painfully aware of G. *grumble* Indeed. The one redeeming quality of your favorite overpriced colo and mine is that when they go to hook up a crossconnect they extend it all the way to the gear without a dozen more tickets, they manage to hook it up correctly the first time, without 1-2 hours of handholding or playing find the port, and without the need to dispatch techs or pay for half an hour of remote hands to roll the damn fibers. :) -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
The price being charged for the public exchange ports is non-trivial Only at the (very few) commercial exchanges. The vast majority are free or of trivial expense. by count of small 10/100 switches or by traffic volume? it costs to build, maintain, and manage an exchange which carries significant traffic. costs get recovered. life is simple. randy
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, ren wrote: 5. Costs. Private peering is expensive, don't let anyone fool you. There is a resource investment in human terms that is rarely calculated properly, I agree with you 100%. Working at a nordic european operator being present at LINX, AMSIX and all the northern europe exchanges my reasoning is this: With IXes you buy one highspeed interface and get lots of peers and you can peer with people you might only exchange a few megabit/s with. Buying loads and loads of OC3s, T3s, OC12 to peer with and purchasing fiber patching to interconnect these just doesnt make sense when you can buy a GE or 10GE interface and get tens or hundreds of peers on that single interface without re-patching or establishing any new fiber connections. We have a very liberal peering policy which makes peering a pure operational decision, being handled by the line organisation. Each peering takes approx 5-10 minutes of someones time and that's it. No meetings of peering coordinators or alike, so those people are freed up to do better things. In a lot of the european exchanges all graphs of all ports on the IX is available to you as a member (or even publically available). If someone runs their port full, you probably know about it. Playing the peering game and trying to increase cost for someone else means you increase your own cost as well. Is that worth it? You have to be pretty big to justify it... -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: it costs to build, maintain, and manage an exchange which carries significant traffic. costs get recovered. life is simple. What is significant traffic? What is the cost? If you have an exchange with let's say 20 people connected to it and they all connect using GE. Running this exchange in an existing facility with existing people, you can easily run it for under $10k per year per connected operator or less as you already have engineers that are on site frequently, you already have a billing department etc. It's when the exchange is being run by a separate entity that needs a marketing department, a well-paid staff of managers, technicians etc that price really goes up. All this to basically manage a simple ethernet switch that needs some patching a couple of times a month at maximum. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
What is significant traffic? What is the cost? If you have an exchange with let's say 20 people connected to it and they all connect using GE. Running this exchange in an existing facility with existing people, you can easily run it for under $10k per year per connected operator or less as you already have engineers that are on site frequently, you already have a billing department etc. It's when the exchange is being run by a separate entity that needs a marketing department, a well-paid staff of managers, technicians etc that price really goes up. All this to basically manage a simple ethernet switch that needs some patching a couple of times a month at maximum. no. in the first case, you're just hiding the incremental costs. eventually, some bean counter is gonna want to recover them, and then folk get quite unhappy. and, there are known issues when a colo or transit provider is the exchange. [ note that i am not talking about small local friendly exchanges. i mean stuff that carries multi-gig. it's like is-is, almost no one runs it, only the few folk who carry most of the internet's traffic. ] randy, who contributes to and peers at the seattle internet exchange
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: no. in the first case, you're just hiding the incremental costs. eventually, some bean counter is gonna want to recover them, and then folk get quite unhappy. What costs are you referring to? You basically need a few hours time per month from engineers and billing department. This for an exchange that has 20 ISPs connected to it. The amount of traffic isn't really a factor, but the one I know of and am part of running carries multi-gigabit. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
Mikael Abrahamsson wrote: On Sat, 3 Jul 2004, Randy Bush wrote: no. in the first case, you're just hiding the incremental costs. eventually, some bean counter is gonna want to recover them, and then folk get quite unhappy. What costs are you referring to? You basically need a few hours time per month from engineers and billing department. This for an exchange that has 20 ISPs connected to it. The amount of traffic isn't really a factor, but the one I know of and am part of running carries multi-gigabit. Does the person that sweeps the floor do so for free? And supply the broom? -- Requiescas in pace o email Ex turpi causa non oritur actio http://members.cox.net/larrysheldon/
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Laurence F. Sheldon, Jr. wrote: Does the person that sweeps the floor do so for free? And supply the broom? The marginal cost of half a rack being occupied by an IX switch in a multi-hundred-rack facility is negiglabe. Yes, it should carry a cost of a few hundred dollars per month in rent, and the depreciation of the equipment is also a factor, but all-in-all these costs are not high and if an IX point rakes in $200k a year that should well compensate for these costs. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 08:47:11AM -0700, Randy Bush wrote: The price being charged for the public exchange ports is non-trivial Only at the (very few) commercial exchanges. The vast majority are free or of trivial expense. by count of small 10/100 switches or by traffic volume? it costs to build, maintain, and manage an exchange which carries significant traffic. costs get recovered. life is simple. I tend to get suspicious when I know the exchange isn't charging enough money to cover its costs. I also don't see a need for a free exchange either. I'm perfectly willing to pay a fair price for the service, and I at least want the BELIEF that I am going to get a certain level of service from the exchange, not but we can't afford... or duhhh?. It seems that most commercial network operators agree, as you rarely see them popping up at joe bob's local alternative new exchange point, even when it is free. The cost for the exchange hardware is really not that much. Just to throw out some numbers, you can snag a new 6509 w/SUP720 and 48-SFP GE for less than $50k with very modest discounts. Admittidly this is relatively new technology compared to most GE exchanges currently deployed, but the pricing a couple years ago was around the same for the Floundry's that everyone deployed, just at a lower density. A successful exchange probably has multiple switches and some 10GE trunks, but with a few customers paying industry average recurring fees this quickly pays for itself. The euro players are really the ones to look to for examples here, US players have been complete failures (especially with multi-site linked exchanges). The guys best positioned to do it are the actual colo operators who already have a technician staff on site, they really only need 1-2 higher level engineers, a support contract for when the switch crashes, etc. The real cost and value of an exchange point is the marketing (i.e. showing up at nanog and giving presentations about it, creating your own peering events, having sales folks promoting the product, etc), not the hardware. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Mikael Abrahamsson wrote: On Sat, 3 Jul 2004, Randy Bush wrote: no. in the first case, you're just hiding the incremental costs. eventually, some bean counter is gonna want to recover them, and then folk get quite unhappy. What costs are you referring to? You basically need a few hours time per month from engineers and billing department. This for an exchange that has 20 ISPs connected to it. The amount of traffic isn't really a factor, but the one I know of and am part of running carries multi-gigabit. This is simply untrue. Whilst it is possible to establish an exchange with minimal cost if it is successful your costs will soon escalate. To provide carrier class service for the worlds top carriers you need to invest in the latest hardware, you need to house multiple switches and odfs in suites, you need to pay a team of engineers to run the exchange 24x7, you need to maintain vendor support agreements. From empirical data this cost is in the order of a few million dollars per year. This may not be a lot of money compared to the annual turnover of the large carriers but eg for a typical exchange $5m between 150 companies is on average about $3k/mo each (of course this will likely be skewed so that the top few companies pay more). If you're exchange is in an already developed location then my observation is that you need to have the above if you are to attract the larger networks which in turn brings in the traffic and noc requirements that see increasing costs. Steve
RE: concern over public peering points [WAS: Peering point speed publicly available?]
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mikael Abrahamsson Sent: Saturday, July 03, 2004 10:22 AM To: [EMAIL PROTECTED] Subject: Re: concern over public peering points [WAS: Peering point speed publicly available?] On Sat, 3 Jul 2004, Laurence F. Sheldon, Jr. wrote: Does the person that sweeps the floor do so for free? And supply the broom? The marginal cost of half a rack being occupied by an IX switch in a multi-hundred-rack facility is negiglabe. Yes, it should carry a cost of a few hundred dollars per month in rent, and the depreciation of the equipment is also a factor, but all-in-all these costs are not high and if an IX point rakes in $200k a year that should well compensate for these costs. -- Mikael Abrahamssonemail: [EMAIL PROTECTED] At the Seattle Internet Exchange a, granted, smaller peering exchange, you have to account for the following costs (and, mind you, this list is not exhaustive). 1) 1 Rack 2) Space for the rack in a secure facility 3) AC for the equipment 4) Power for the equipment (including line and UPS) 5) Fiber and Copper runs to the facility for cross-connects 6) Terminations of (5) 7) OM of space and gear 8) Layer 8 and 9 negotiation of (1) through (7) to keep costs down. That's not a trivial set of expenses, particularly when there are limitations in place to recovering costs via non-cash methods, such as advertising the hosting of the exchange. Thankfully, there is some altruism on the behalf of several parties that allow the exchange to continue providing zero cost connections to participants. I hardly think the cost of their time and effort is marginal. Mike NoaNet
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Stephen J. Wilcox wrote: This is simply untrue. Whilst it is possible to establish an exchange with minimal cost if it is successful your costs will soon escalate. To provide carrier class service for the worlds top carriers you need to invest in the latest hardware, you need to house multiple switches and odfs in suites, you need to pay a team of engineers to run the exchange 24x7, you need to maintain vendor support agreements. IXes are not for top carriers, they're for the small and middle players, and in some cases for the top players to talk to smaller players. IXes is a way to cheaply exchange traffic. It's better to establish two IX switches and run them with 99.9% availability than to have a single IX switch and aim for 99.999%. If you're exchange is in an already developed location then my observation is that you need to have the above if you are to attract the larger networks which in turn brings in the traffic and noc requirements that see increasing costs. If you're already an operator or colo facility owner, you already have all of that, which makes the cost of running an IX much less than if you're a separate entity who have to set up all these facilities. I work in an environment where IXes are readily available in all major metropolitan areas where we are, and they don't cost an arm and a leg and fiber is cheap and readily available, so we try to establish everywhere. This brings the impact of a single IX being down to very negligable, so we definately don't need 99.999%. Off the top of my head, I'd estimate that the cost of being present at an exchange here is around $1-5k per gig per month (including router port, fiber connection and IX exchange fee). We run these at approx 50% utilisation so the price per megabit is $5-10/megabit per month. This also adds a lot of reduced latency from our customers to our competitors customers which is very appreciated, it also cuts down on long-haul costs. If an IX costs $50-100k a year for a gig it tilts the whole equation, so I can understand if a lot of people don't like them if that's the cost of being connected. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
RE: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Michael Smith wrote: 1) 1 Rack 2) Space for the rack in a secure facility 3) AC for the equipment 4) Power for the equipment (including line and UPS) This can be had for approx $300-1000 a month in my market. 5) Fiber and Copper runs to the facility for cross-connects 6) Terminations of (5) This is carried on a per connection basis in my market. 7) OM of space and gear $50-100k over three years isn't that much. 8) Layer 8 and 9 negotiation of (1) through (7) to keep costs down. I'd say that the time spent in negotiations is wasted, manpower is too expensive compared to the costs involved. Thankfully, there is some altruism on the behalf of several parties that allow the exchange to continue providing zero cost connections to participants. I hardly think the cost of their time and effort is marginal. In the big picture it's marginal. Asking someone to patch a cable is a 10 minute job and the patch cable costs perhaps 30-50 dollars. Handling an invoice for this job is a major cost in the equation so yes, altruism is great. We gathered players that already had engineers, already had billing departments, already had all of the above you were referring to and get everybody to agree on a way to cooperate. The marginal costs for everybody to establish 5 PoPs and interconnecting them was quite low and since there are no billing being done between participants, that cuts down on paperwork as well. It's like a car pool. If everybody is going to bill everybody it's going to be a big operation. If you just agree to drive every fourth day and carry your own costs, everybody is better off. I realise from everybody who answered that we live in different markets and do things differently. I just think you're making it too advanced and that increases cost until public IXes stop to make sense. Keep it simple. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
i look forward to my next trip to sweden, where i expect many nice free lunches randy
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: i look forward to my next trip to sweden, where i expect many nice free lunches If you start working in a resturant, you can probably expect that. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
i look forward to my next trip to sweden, where i expect many nice free lunches If you start working in a resturant, you can probably expect that. but you seem to think they are served in exchange points, and not just to those that run them, but to all comers. very cool. sad to say, we're past 1999 now. out here in the free world (and those countries we bomb and/or invade[0]) folk seem to want us to pay for what we eat. bummer, eh? randy -- [0] - bumber sticker of the week We're making enemies faster than we can kill them!
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 10:57:20AM -0700, Michael Smith wrote: At the Seattle Internet Exchange a, granted, smaller peering exchange, you have to account for the following costs (and, mind you, this list is not exhaustive). 1) 1 Rack 2) Space for the rack in a secure facility 3) AC for the equipment 4) Power for the equipment (including line and UPS) 5) Fiber and Copper runs to the facility for cross-connects 6) Terminations of (5) 7) OM of space and gear 8) Layer 8 and 9 negotiation of (1) through (7) to keep costs down. That's not a trivial set of expenses, particularly when there are limitations in place to recovering costs via non-cash methods, such as advertising the hosting of the exchange. Thankfully, there is some altruism on the behalf of several parties that allow the exchange to continue providing zero cost connections to participants. I hardly think the cost of their time and effort is marginal. Which means that SIX's costs would be completely covered by charging each member with a GigE port $1k/mo. I would rather pay them the $1k/mo with the expectation that they will be able obtain quality hardware (which btw doesn't necessarily mean running to their favorite vendor and asking for the most expensive product available), provide reliable service, handle growth, etc. I would not however, pay them $14k/mo for the same service. I count 68 active participants on the SIX website. I won't venture a guess as to how many have GigE ports, and a few are connected from PAIX, etc, but I would bet that there is more than enough business available to cover the costs of intelligent spending. You could probably still give away FastE ports for free, and pretty much assume that any major ISP who can afford the GigE port and sees value in connecting with the smaller guys will go ahead and pay for it. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
beware. six is funny. it's in seattle's carrier hotel, the westin, 32 floors of racks, more colo providers than fleas on a dawg, and very very low inter-suite fiber rates from the building owners. so, though the six does have a core, it is also kinda splattered into switches all over the building; with ease of connection and low cost being achieved at the expense of reliability. and costs are distributed along with the six infrastructure. so colo provider A may have a switch and charge $a to access it, while colo provider B may charge $b, where $b != $a. for a small local exchange this is ok, even cool. i would not want to do similarly in virginmania or palo attitude, and i would not join the six if i was a major player (only a research rack is on the six). my internal indirect costs would not be worth the traffic shed. randy
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, 3 Jul 2004, Randy Bush wrote: but you seem to think they are served in exchange points, and not just to those that run them, but to all comers. very cool. sad to say, we're past 1999 now. out here in the free world (and those countries we bomb and/or invade[0]) folk seem to want us to pay for what we eat. bummer, eh? The weird thing is that I (and partners) have been running an IX wth 4 nodes since 2001 with the business model I have mentioned and as far as I can calculate, we have at least made break-even. At $5k a year for FE and $10k a year for GE and letting the ISP provide their own access to the IX via whatever means they have available, it's possible to run an IX if you just want to provide the IX L2 unicast service and not have a lot of other services around. We calculated that we needed three customers per PoP and we've had more than that. The initial investment in switches was approx $50k per PoP. Running L2 switches is quite simple, I don't see what all the fuss is about. If the above model doesnt work in your area, though luck for you, guess you have to pass on the added cost to your customers. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: concern over public peering points [WAS: Peering point speed publicly available?]
On Sat, Jul 03, 2004 at 01:39:03PM -0700, Randy Bush [EMAIL PROTECTED] wrote: building owners. so, though the six does have a core, it is also kinda splattered into switches all over the building; with ease of connection and low cost being achieved at the expense of reliability. Though that's true, the SIX has been extremely reliable: one unscheduled core outage in the last 3 years (about 30 minutes due to power loss). In one other case, an extension switch (7 peers) was disconnected for about 30 minutes to troubleshoot a potential problem. Peer-operated extension switches have also been very reliable. Most are above 99.9% availability including scheduled maintenance and 99.99% for unscheduled problems. The SIX's staffed 24x7 NOC lets peers treat it like any other carrier relationship, with one phone number to report a problem. Often the ops staff at national networks never know the SIX is non-profit or donation-supported. Peers of all sizes seem happy with the reliability. Everyone has open-posting mailing lists and an annual opportunity to elect the Board of Directors, so there is recourse if circumstances change. Cheers, Troy (SIX janitor)
Re: concern over public peering points [WAS: Peering point speed publicly available?]
let's just say that my experience is not all that reliable. i i suspect it varies greatly between colo/sub-switch providers. but considering the cost, i ain't got no complaints. qed. randy
