Flast model users (was: Re: Lessons from the AU model)
Dear Mark, - Original Message - From: "Mark Newton" <[EMAIL PROTECTED]> Do you think Australian ISPs haven't tried to offer US-style flat-rate services? Of course they have. And they get destroyed in the marketplace. Here's the thing that metering gives you: it stratifies the marketplace. It gives you two classes of customer. One class is customers who know they can live painlessly within the boundaries of whatever quotas you're offering. They don't complain, they just pay their flat monthly bill every month and get on with their lives. The other class is customers who do so much P2P that the imposition of quotas is a painful and unwelcome experience. They whinge and bitch loudly about how awful their ISP is, and migrate en-masse towards whichever ISPs are providing "unlimited" services. The only people who truly care about "unlimited" are the ones who know they can't live within any limits. That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that. I can't agree with your definition of users. In community that has been also in a bit isolated (czech republic), there has been monopoly for cables/lines to Western europe till end of the 2K. Then after deliberalisation (and later sale of state owned ILEC) many new players entered the market and prices felt down. So it seems like Australia is somewhere in our trace. In meantime, some companies tried to start metered services. Some failed, some survived. But no one from surviving based his metering on payments on gigabytes. Most of them are working to soft limits and aggregation. Just because there is a problem that user cannot understand what is he consuming. You can meter water or minutes consumed on telephone. But if you are tuning up your favorite newschannel over Internet, do you know how many MB you are transferring each minute? Other problem is unsolicited traffic. Most of "plain users" have their PCs not well secured. And worms and botnets can create same traffic as P2P. Same situation for application that are making money over your business (i.e. Joost or Skype). You can argue that people cannot understand eletricity consumption and they are paying for usage, but - someone has to fire a coal or broke uranium atoms to develop elettricity. Are ISPs developping Mbytes or are they paying to Yahoo!, Google or other ICP? For me, raising this discussion from ISP community will lead to discussion on net neutrality from other (unwanted) side. May AT&T pay to Yahoo! or Google for data that they are selling to their customers? Regards Michal P.S: To not be completelly off-topic. For last two weeks I've been visiting New Zealand (nomadic access). And last few days I'm in Australia. Let me tell you that it is unblievable that these two countries are on the same submarine cable systems. NZ looks from nomadic perspective as US or NL, but I've been more lucky to get connectivity in Beijing that in Melbourne neighborhood :-(
Re: Lessons from the AU model
$quoted_author = "Tom Vest" ; > > Occasional rhetorical indulgences notwithstanding, I'm a pragmatist; an > ever-rising upper limit that 99% of the population never ever notices is > not much of a limit. Sure it is. By knowing that no-one sharing the backhaul to the DSLAM at my CO can afford to do line rate 24x7 makes me sleep better at night. By using an ISP with sane limits I've never noticed performance degradation, even during peak periods. > However I've rarely (actually, before now, *never*) heard the AU/NZ > situation described thusly I must be spending too much time with the > wrong 2% I guess. The wrong 2% of what? :-) If you want to see the flat-rate churning horde in all their glory visit whirlpool.net.au, otherwise affectioninatly known as "whingepool" because of the bitching'n'whining every time an ISP goes under or attends ECONOMICS 101 and brings in sustainable limits. > And I've yet to hear how one will be credibly define or sustainably (and > legally) maintain such escalating limits. Simon's post pretty much sums up the kind of maths that justifies them. http://www.merit.edu/mail.archives/nanog/msg05636.html Legal? What's legality have to do with this discussion? cheers marty -- "Multiple coffee suppliers feeding into a Redundant Arrangement of Independent Dispensers, to further reduce the chance of uncaffeinated downtime and increase Mean Time To Drowsiness." --Steve VanDevender alt.sysadmin.recovery - <[EMAIL PROTECTED]>
Re: Lessons from the AU model
[EMAIL PROTECTED] (Geoff Huston) writes: > Which is a roundabout way of saying that I'm very sceptical of Tom's > exponential optimism in this particular area of infrastructure investment > :-) the internet often hasn't made good economic sense. consider the chewage and swallowage of resources by modems, which eventually reached the limit where every iota of voicegrade on a circuit was in use, causing telco's to lose profit because they had an overcommit model and whereas they could compress 98% of a voice call they could not compress any of a modem call. flat rate connectivity (insensitive to distance or kilosegments or etc) is only practical under massive overprovisioning. we need moore's law to keep on going forever, and we need similar speeds of advance in optics and in power utilization. companies who can't massively overprovision will just continue to be shouldered aside by companies who can. new capital will keep on entering the market, and win or lose based on some disruptive approach, and then the winners will get shouldered aside a generation or two later. it's 2008, and no part of this should sound like news or insight to anybody. what i like about non-flat models like the one described as working for NZ and AU is that it will keep truly worthless flows off the network. finally there's a reason not to mindlessly share everything with everybody everywhere. (which is the only part of the equation that free market capitalism can't otherwise solve.) -- Paul Vixie
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Tom Vest wrote: Okay I concede that point; competition within markets with only metered service options can be just as or even more vigorous then competition within unmetered markets. But competition between metered and unmetered markets tends to reward the latter, and competition within mixed markets tends to reward the latter. Moreover, to whatever degree that metering reduces usage, and usage is related to innovation, the unmetered markets are likely to grow and evolve faster. Maybe flat-rate access is "objectively" unsustainable in some markets, regardless of what (potentially self-interested) proponents claim. Maybe it is sustainable, even in the "long run", despite what (potentially self-interested) critics claim. One thing is certain: to date, only a handful of companies (Internet, wireless voice, or POTS) have *ever* stepped up to any flat-rate service voluntarily. The vast majority of flat-rate providers in business today were just as skeptical, and just as implacably opposed to the idea, right up to the point that competition forced them (and permitted us) to discover that they were wrong. At least that's what happened in Japan, in the UK, and yes in the US and many other places too. If you beleive it is inevitable, why do you care what other people do in the mean time. If you are correct, they will eventually loose and go away. Let them try it and see what happens. Why pre-judge the results?
Re: Lessons from the AU model
On Jan 22, 2008, at 3:01 AM, Mark Newton wrote: On 22/01/2008, at 3:59 PM, Tom Vest wrote: When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates. I believe that someone will be able to "build" (i.e., finance) more, when/where more is required. Faith-based network rollouts. Neat :-) Just faith in the "rationality" of a system with lots of independent feedback loops. If your faith has conflicting tenets, e.g., it compels you to believe that banks will demand, and you will be able to credibly promise, bubble-era returns on gross facilities investments, then I think we can chalk the rest up to religious differences and call it a day ;-) Sounds like change is afoot however; I wonder how closer to flat- rate AU will come after one or two new cables are completed... ? My guess? There'll continue to be industry-wide acceptance of the notion that a limit you never reach and don't care about is indistinguishable from no limit at all, and quotas will continue to steadily rise, stimulated by the lowering of the costs of supply, so that more and more people will never reach them and never care about them. Occasional rhetorical indulgences notwithstanding, I'm a pragmatist; an ever-rising upper limit that 99% of the population never ever notices is not much of a limit. However I've rarely (actually, before now, *never*) heard the AU/NZ situation described thusly I must be spending too much time with the wrong 2% I guess. And I've yet to hear how one will be credibly define or sustainably (and legally) maintain such escalating limits. _Most_ people couldn't care less about whether their monthly limit is 100 Gbytes or 150 Gbytes. But there are a small number of people who will be unhappy with any number smaller than 6 Tbytes, which is roughly the amount you can download on a 24 Mbit/sec ADSL2+ access tail if you run it flat-out 24x7 for a whole month. If new cable systems lower the cost of getting across the Pacific by a factor of 2, then 24 Mbit/sec of transpacific for that customer will _still_ cost us about $3000 per month, so you won't see us flocking to service those sorts of customers. You don't know what the "cost" is today; you only know the price. The implications left as an exercise for the reader... There is simply no way that any provider in this marketplace is going to offer to service any customers on those terms. To get Australian providers to sell "unlimited" access you'd need the price of crossing the Pacific to drop by two orders of magnitude. ... Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month. Here's another dynamic you missed. The coexistence of flat-rate and metered access creates two tiers of regional markets: the flat rate ones that generate lots of content and service innovations, attract lots of talent and FDI, and over time come to occupy an increasingly central position in the evolving global L2/L3 topology -- and the ones that don't, whose native innovators and content providers prefer to ship out or offshore production to the former. Metered access exacts a price at the national level. I don't doubt that at all, and the USA has certainly done very well out of it. But how sustainable is it in the long term? You guys are about to have some serious economic challenges that are going to affect us too, but nowhere near as much as they'll affect you. I don't doubt that a difference in Internet access pricing policies has benefited your economy, that much is completely obvious. ok. But if you're going to get all macro and holistic on me (actually that's my job) all of a sudden, are you prepared to assert that the benefit it has brought to your economy outweighs the other consequences that your ways of doing business have been creating? I wouldn't attempt to justify anything (and folks that know me will attest that they often hear something like the opposite), but even if I did, even for purely rhetorical amusement, I would start by pointing out that flat-rate access doesn't have any relationship to anything else you might be obliquely alluding to here. Being macro doesn't imply or necessitate being a dope ;-) Because from where I'm sitting, it looks like the native innovators are in India at the moment, Hard to argue with that. Since "constructive" innovation is a gross numbers game (e.g., education * addressable population), demographics ultimately drives a lot of dynamics -- once other bottlenecks are relieved. and the global L2/L
RE: Lessons from the AU model
We've figured our customer base ranges between 8 to 12 kbps/customer. Frank -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alastair Johnson Sent: Tuesday, January 22, 2008 4:09 AM To: nanog@merit.edu Subject: Re: Lessons from the AU model Mark Newton wrote: > Despite the best efforts of some people to run their broadband > access at line rate, residential broadband is very much a > "CIR + burst" kind of service. All of our customers can burst > to line rate (they're paying for it, so they should be able to > get it). None of our customers can burst at line rate 24x7 for > a month without paying for it. You can work out the CIR by > dividing the number of bits in the quota by the number of > seconds in a month. Indeed. If you look at New Zealand, a very similar economic model to Australia (except less population and a much bigger density problem), there are regulated wholesale products[1,2,3] that offer a 32Kbps CIR per subscriber, and linerate PIR. 32Kbps working out to approximately 10GB per month, you can guess what the most common subscriber data cap is - and surprisingly few actually exceed it, although it has definitely gone up. Incidentally, the incumbent in NZ launched a flat rate DSL package. It did not go well, and ultimately cost them several million dollars in subscriber refunds. Perhaps some of the guys posting on this thread (Mark? MMC?) would be able to provide an average subscriber bandwidth (in GB or Kbit/s) use of their subscriber base. Break it down by <10G, >40G type accounts? aj [1] http://www.comcom.govt.nz/IndustryRegulation/Telecommunications/Wholesale/Ov erview.aspx for what the regulator is doing [2] http://www.telecom.co.nz/content/0,8748,205743-204225,00.html?nv=tpd [3] http://www.telecom.co.nz/content/0,8748,204215-204225,00.html?nv=sd
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Mikael Abrahamsson wrote: Your analogy is halting, but that's to be expected. I certainly wouldn't want to pay more for the landlord to install metering everywhere. There is much overhead in metering and billing on that. I suspect you have never been a landlord or needed to respond to tenants.
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Sean Donelan wrote: If there was one tenant that left the hot water running 24 hours, 7 days a week; so other tenants complained they didn't get enough hot water. One That has never happened to me. We have good enough infrastructure that one tenant filling up their hot water bath doesn't deplete the infrastructure of "hot water production" in my building. I seriously doubt anyone would notice me running hot water 24/7, because the infrastructure is able to handle that. No, everybody can't do it, but if I need to for a couple of hours, it works. tenant plugged in maximum wattage heaters on every circuit and left them on high 24 hours a day; left the television volume turned up to the maximum 24 hours a day; and so forth. I know people who run servers in their dorms due to this. It might go away, power is easy to meter. If you were the neighbor of such a tenant in a building, would you be pleased that your monthly fees were being increased or that one tenant was using all the hot water and generating a lot of noise all day and all night? Or might you complain to the landlord about those problems. Your analogy is halting, but that's to be expected. I certainly wouldn't want to pay more for the landlord to install metering everywhere. There is much overhead in metering and billing on that. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Lessons from the AU model
On 22/01/2008, at 7:30 PM, Mikael Abrahamsson wrote: I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that. It isn't hard. For a start, there are only one or two providers where customers ever get excess bills, so we aren't really talking about "billing when a person is being DDOS'ed", we're talking about "rate- limiting after a person has been DDOS'ed." On the rare occasion when it happens, the ISPs who aren't bastards are usually understanding enough to back out measurements which are DDOS related. Whether or not your ISP is a bastard is one of the factors you'd use to determine which ISP is offering a package that fits your needs in a competitive environment. In my observation, customers who are repeatedly DDOS'ed are usually doing something to provoke it. I don't think I've ever seen an instance where a DDOS'ed customer has had a repeat occurrance after that datapoint is illuminated for them. - mark -- Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Mikael Abrahamsson wrote: I know of places in my nick of the world where all those are flat-rate. When the usage difference is small enough, metering is not effective. Ahh, the key phrase is "usage difference is small enough." Typical dorm here includes power, water, (gas is usually not used, but most of the places that have gas charge ~ USD15 per month per apartment for gas, flatrate), and heating. Basic cable included in rent. I also know of quite a few regular apartments that have this model. In my apartment I pay for power. Water, heating and basic cable is included in the monthly fee. But what if the usage differences were large. If there was one tenant that left the hot water running 24 hours, 7 days a week; so other tenants complained they didn't get enough hot water. One tenant plugged in maximum wattage heaters on every circuit and left them on high 24 hours a day; left the television volume turned up to the maximum 24 hours a day; and so forth. Should the response from the landlord to install bigger hot water heaters, more electrical circuits, rebuild all the walls with increased sound proofing, and pass the cost of all those improvements to all the tenants in the building? Or, would the landlord consider asking one tenant with unusually high demands to leave or raise that just that tenant's rates. If you were the neighbor of such a tenant in a building, would you be pleased that your monthly fees were being increased or that one tenant was using all the hot water and generating a lot of noise all day and all night? Or might you complain to the landlord about those problems.
RE: Lessons from the AU model
Or even Blue Security. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ben Butler Sent: 22 January 2008 10:26 To: nanog@merit.edu Subject: RE: Lessons from the AU model Hi, Regarding Dos filtering, I guess that really depends on whether we are talking about completing the attack and filtering in upstream transits, or, filtering source / traffic classification within the AS keeping the destination alive throughout the attack and utilising WAN/Transit bandwidth in the process. For me personally, if the first scenario is picked up under the horizon of the 95% bill then its a non issue. If the customer wants to ride it out themselves with no filtering from us, then it is going to appear on their port and hence burst bill for the month. And if the customer wants us to upstream source/classify filter traffic to give them a sanitized feed while we still incur the traffic - well they either pay or get turned off. Personally I don't want a Blue Solutions and if the customer were to think I am going to filter 24gbps of DoS delivering them a few megs of cleaned up traffic and that is all they pay for Well, they will be getting a P45. Dos filtering is expensive and value add and people need to pay fair price for things in this world, life is not free / cheap. A lack of imagination or focused usage from the IPTv / download brigade - does not mean something has lower value just that their business model is flawed and they cant afford the tools for the job + then keep all the advertising revenue to themselves. Why should access providers be giving away Ferraris for less than the manufacture cost of Maxi's (a really naff old UK car - http://en.wikipedia.org/wiki/Austin_Maxi) There is a UK ISP offering a LLU DSL service (line rental + internet + telephone service) for less than the monthly cost of the copper line rental from BT - to try and build market share to "pay" for that expensive DSLAM rollout by "buying" market share - I do at times wish the FDs / marketing / sales people at some of these companies would go and sit some basic business studies / accountant / economics classes. My 2c. Ben -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alastair Johnson Sent: 22 January 2008 09:55 To: nanog@merit.edu Subject: Re: Lessons from the AU model Mikael Abrahamsson wrote: > Some claim that metering is 50% of cost in the telco industry, and I > have no reason to doubt that. Staying out of metering saves money on > all levels, less complex equipment, less supportcalls, less hassle > with billing. I have to agree with this, although the figure is trending downwards. Certainly one situation I have seen with ~$200mm in broadband metering revenue was spending $80m-ish a year on the various platforms that managed metering/rating/billing and the ops cost that went with it. The flipside is that I can see products (BRAS/BNG and their associated control plane solutions) being developed, launched, and marketed right now that make this much easier to manage. If the telephants adopt this, then it will reduce their billing cost substantially - although it adjusts (disrupts?) their traditional messy OSS/BSS. > I am also hesitant regarding billing when a person is being DDOS:ed. > How is that handled in .AU? I can see billing being done on outgoing > traffic from the customer because they can control that, but what > about incoming, the customer has only partial control over that. In my experience (NZ & AU) inbound DDoS attacks are usually waived by the service provider. This may not apply with all ISPs, but when I drove an ISP we did try and protect customers from that form of bill-shock. aj.
RE: Lessons from the AU model
Hi, Regarding Dos filtering, I guess that really depends on whether we are talking about completing the attack and filtering in upstream transits, or, filtering source / traffic classification within the AS keeping the destination alive throughout the attack and utilising WAN/Transit bandwidth in the process. For me personally, if the first scenario is picked up under the horizon of the 95% bill then its a non issue. If the customer wants to ride it out themselves with no filtering from us, then it is going to appear on their port and hence burst bill for the month. And if the customer wants us to upstream source/classify filter traffic to give them a sanitized feed while we still incur the traffic - well they either pay or get turned off. Personally I don't want a Blue Solutions and if the customer were to think I am going to filter 24gbps of DoS delivering them a few megs of cleaned up traffic and that is all they pay for Well, they will be getting a P45. Dos filtering is expensive and value add and people need to pay fair price for things in this world, life is not free / cheap. A lack of imagination or focused usage from the IPTv / download brigade - does not mean something has lower value just that their business model is flawed and they cant afford the tools for the job + then keep all the advertising revenue to themselves. Why should access providers be giving away Ferraris for less than the manufacture cost of Maxi's (a really naff old UK car - http://en.wikipedia.org/wiki/Austin_Maxi) There is a UK ISP offering a LLU DSL service (line rental + internet + telephone service) for less than the monthly cost of the copper line rental from BT - to try and build market share to "pay" for that expensive DSLAM rollout by "buying" market share - I do at times wish the FDs / marketing / sales people at some of these companies would go and sit some basic business studies / accountant / economics classes. My 2c. Ben -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alastair Johnson Sent: 22 January 2008 09:55 To: nanog@merit.edu Subject: Re: Lessons from the AU model Mikael Abrahamsson wrote: > Some claim that metering is 50% of cost in the telco industry, and I > have no reason to doubt that. Staying out of metering saves money on > all levels, less complex equipment, less supportcalls, less hassle > with billing. I have to agree with this, although the figure is trending downwards. Certainly one situation I have seen with ~$200mm in broadband metering revenue was spending $80m-ish a year on the various platforms that managed metering/rating/billing and the ops cost that went with it. The flipside is that I can see products (BRAS/BNG and their associated control plane solutions) being developed, launched, and marketed right now that make this much easier to manage. If the telephants adopt this, then it will reduce their billing cost substantially - although it adjusts (disrupts?) their traditional messy OSS/BSS. > I am also hesitant regarding billing when a person is being DDOS:ed. > How is that handled in .AU? I can see billing being done on outgoing > traffic from the customer because they can control that, but what > about incoming, the customer has only partial control over that. In my experience (NZ & AU) inbound DDoS attacks are usually waived by the service provider. This may not apply with all ISPs, but when I drove an ISP we did try and protect customers from that form of bill-shock. aj.
Re: Lessons from the AU model
Mark Newton wrote: Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month. Indeed. If you look at New Zealand, a very similar economic model to Australia (except less population and a much bigger density problem), there are regulated wholesale products[1,2,3] that offer a 32Kbps CIR per subscriber, and linerate PIR. 32Kbps working out to approximately 10GB per month, you can guess what the most common subscriber data cap is - and surprisingly few actually exceed it, although it has definitely gone up. Incidentally, the incumbent in NZ launched a flat rate DSL package. It did not go well, and ultimately cost them several million dollars in subscriber refunds. Perhaps some of the guys posting on this thread (Mark? MMC?) would be able to provide an average subscriber bandwidth (in GB or Kbit/s) use of their subscriber base. Break it down by <10G, >40G type accounts? aj [1] http://www.comcom.govt.nz/IndustryRegulation/Telecommunications/Wholesale/Overview.aspx for what the regulator is doing [2] http://www.telecom.co.nz/content/0,8748,205743-204225,00.html?nv=tpd [3] http://www.telecom.co.nz/content/0,8748,204215-204225,00.html?nv=sd
Re: Lessons from the AU model
Mikael Abrahamsson wrote: Some claim that metering is 50% of cost in the telco industry, and I have no reason to doubt that. Staying out of metering saves money on all levels, less complex equipment, less supportcalls, less hassle with billing. I have to agree with this, although the figure is trending downwards. Certainly one situation I have seen with ~$200mm in broadband metering revenue was spending $80m-ish a year on the various platforms that managed metering/rating/billing and the ops cost that went with it. The flipside is that I can see products (BRAS/BNG and their associated control plane solutions) being developed, launched, and marketed right now that make this much easier to manage. If the telephants adopt this, then it will reduce their billing cost substantially - although it adjusts (disrupts?) their traditional messy OSS/BSS. I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that. In my experience (NZ & AU) inbound DDoS attacks are usually waived by the service provider. This may not apply with all ISPs, but when I drove an ISP we did try and protect customers from that form of bill-shock. aj.
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Mikael Abrahamsson wrote: > I am also hesitant regarding billing when a person is being DDOS:ed. How > is that handled in .AU? I can see billing being done on outgoing traffic > from the customer because they can control that, but what about incoming, > the customer has only partial control over that. DOS's against home customers arn't *that* common, certainly those that last long enough to hit bandwidth quota don't happen very often. In the past when you paid for going over your quota people did get $5000 bills for their home accounts. The terms and conditions made the customer responsible for it fullstop. It was their job to monitor their usage. The "throttle on cap" method tends to fix the problem. The customer does a huge amount of traffic unexpectantly so they just get slower Internet. Repeat until customer learns to not leave p2p programs running all night or let junior hang around the wrong channels on IRC. Usually they will get an email when they are at 80% of their limit or something which helps more. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Tom Vest wrote: > But even assuming you manage to define a "reasonable" cap, how will > you defend it against competitors, and how will you determine when & > how to adjust it (presumably upwards) as the basket of "typical" user > content and services gets beefier -- or will that simply tip more and > more people into some premium user category? Seriously Tom it's not *that* hard and like we've been saying plenty of ISPs in many countries manage to do it. The different companies just play around until your profit, costs and income all balance nicely In NZ and Aus as the costs of bandwidth have decreased (and the demand has increased) then the bandwidth quotas have tended to go up. Lets say that a customer costs around $25 per month in last mile charges, staff, billing, marketing, profit for a 6Mb/s DSL. In the US right now you spend $5 on marginal bandwith usage which at $10 per MB/s gets you 150GB across the month. In Australia where the bandwidth cost is closer to $200 per MB/s per month that $5 only gets you around 8GB. Pricing flat rate will either put you out of business or cost so much that you not get 90% of customers. So in Australia you'll do a $30 cheap account with a 5GB/month quota, a $40 account with a 15GB Quota and a $60 account with a 40GB/month quota. This keep you bandwidth cost about the same and allows you to capture low end customers for $30 as well as heavier users at $60. Cheap entry level option and heavier users can pay more to get more. In the US 150GB is more than 90-something percent of users do and 6MB/s line speeds ( mostly) keeps the heavier users from going high enough to screw the average above that. So you keep a simple flat pricing scheme because that is easier to market and makes you more money. Just like helpdesk calls are usually free even though some users use up hundreds of dollars worth of them per month. On the other hand, image a few years down the track and you are at a US provider with $5 per Mb/s per month transit costs, most of your customers have 100Mb/s connections and : the bottom 30% of your customers average 0.25 TB / month = $4 / customer The next 40% of your customers average 1 TB / month = $15 / customer the top 30% of them average 3 TB / month = $45 / customer So you average bandwidth cost is around $20 per customer. Options are: 1. Increase prices to $45 per month and keep flat rate 2. Introduce tiered accounts 3. Traffic shap until bandwidth costs drop enough. Now right now option 3 seems to be common which sort of indicates that bandwidth usage by home customers *is* a problem. In many cases the choke point is at the last mile but it still doesn't really change the numbers. Providers have a budget to spend per customer on bandwidth, when they start to exceed that then something has to give. Of course with a bit of luck the cost of providing bandwith will keep falling as fast or faster than the average customer demand. Personally I doubt that long term home demand will exceed 30Mb/s or 10TB per month ( around 1 HDTV channel) on average. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Mark Newton wrote: Power is metered. Water is metered. Gas is metered. Heating oil is metered. Even cable-TV is packaged so that you pay more if you want to use more channels... I know of places in my nick of the world where all those are flat-rate. When the usage difference is small enough, metering is not effective. Typical dorm here includes power, water, (gas is usually not used, but most of the places that have gas charge ~ USD15 per month per apartment for gas, flatrate), and heating. Basic cable included in rent. I also know of quite a few regular apartments that have this model. In my apartment I pay for power. Water, heating and basic cable is included in the monthly fee. Some claim that metering is 50% of cost in the telco industry, and I have no reason to doubt that. Staying out of metering saves money on all levels, less complex equipment, less supportcalls, less hassle with billing. I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Lessons from the AU model
On 22/01/2008, at 3:59 PM, Tom Vest wrote: When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates. I believe that someone will be able to "build" (i.e., finance) more, when/where more is required. Faith-based network rollouts. Neat :-) Sounds like change is afoot however; I wonder how closer to flat- rate AU will come after one or two new cables are completed... ? My guess? There'll continue to be industry-wide acceptance of the notion that a limit you never reach and don't care about is indistinguishable from no limit at all, and quotas will continue to steadily rise, stimulated by the lowering of the costs of supply, so that more and more people will never reach them and never care about them. _Most_ people couldn't care less about whether their monthly limit is 100 Gbytes or 150 Gbytes. But there are a small number of people who will be unhappy with any number smaller than 6 Tbytes, which is roughly the amount you can download on a 24 Mbit/sec ADSL2+ access tail if you run it flat-out 24x7 for a whole month. If new cable systems lower the cost of getting across the Pacific by a factor of 2, then 24 Mbit/sec of transpacific for that customer will _still_ cost us about $3000 per month, so you won't see us flocking to service those sorts of customers. There is simply no way that any provider in this marketplace is going to offer to service any customers on those terms. To get Australian providers to sell "unlimited" access you'd need the price of crossing the Pacific to drop by two orders of magnitude. Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month. Here's another dynamic you missed. The coexistence of flat-rate and metered access creates two tiers of regional markets: the flat rate ones that generate lots of content and service innovations, attract lots of talent and FDI, and over time come to occupy an increasingly central position in the evolving global L2/L3 topology -- and the ones that don't, whose native innovators and content providers prefer to ship out or offshore production to the former. Metered access exacts a price at the national level. I don't doubt that at all, and the USA has certainly done very well out of it. But how sustainable is it in the long term? You guys are about to have some serious economic challenges that are going to affect us too, but nowhere near as much as they'll affect you. I don't doubt that a difference in Internet access pricing policies has benefited your economy, that much is completely obvious. But if you're going to get all macro and holistic on me all of a sudden, are you prepared to assert that the benefit it has brought to your economy outweighs the other consequences that your ways of doing business have been creating? Because from where I'm sitting, it looks like the native innovators are in India at the moment, and the global L2/L3 topology that's been built over the last five years has been centering itself on Singapore, Hong Kong and Tokyo, and most of the high-growth parts of the US economy have been based on unsustainable debt rather than actual, existing money, and despite your claims of high growth my standard of living in Australia is at least as good as yours. So don't be too hasty to tell me all about the benefits of doing business in the American way. Not yet, anyway. :-) Where I come from household water is already metered, so I'm not sure what you're talking about :-) Agreed, but I almost never hear my water vendor claiming that my rates need to be increased so they can cross-subsidize a new, separate national plumbing platform. That's exactly what my water utility is claiming. Australia is in the worst drought in recorded history, and all the water utilities are saying their rates are going to rise so that they can afford to build desalination plants and feed them with electricity. I'm tellin' ya, man, the parallels to this discussion are eerie :-) The only people they'll piss off will be the 2% of customers they don't want in the first place. Hardly fertile ground for a revolution. That almost sounds unobjectionable, I'll grant you -- at least in currently unmetered, high demand markets; everywhere else the bandwidth demand for any/every customer segment is nothing more than an artifact of whatever metered pricing plan is currently in place. So the reasonableness of the bandwidth cap level is itself contingent on the reasonableness of the metered plan. The metered plan will be reasonable, as dictated by all the competit
Re: Lessons from the AU model
On Tue, 22 Jan 2008, Mark Newton wrote: That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that. Ah, you've discovered our secret plan. If the US stops being the flat-rate P2P seeder to the world, what is the rest of the world going to do? Or is it going to topple countries one by one into metering? Are the flat-rate Swedish ISPs are going to get stuck footing the bill for the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers for the world :-)
Re: Lessons from the AU model
Tom Vest wrote: So if they don't have a billion or so dollars stored away somewhere, they're selling below replacement value. With very few exceptions there's no "they"; the old "they" is gone, the new "they" didn't take over until fairly recently, didn't bankroll the original construction, and isn't bearing the financing costs of that construction. Round this neck of the woods we call em "banks" and they've generally been around for a few centuries and some of them, current financial shifts notwithstanding, expect to be around for a while yet! It's these folk that have the task of pricing risk and the greater the risk the more expensive the capital, of course. The situation Mark was alluding to was a persistent theme of the submarine cable conferences a couple of years back, where folk were telling each other that the retail IP market got itself hooked on access to wholesale submarine transit assets at distressed prices and the proposition was being made that this business model of flogging off bankrupt assets for cents in the dollar simply didn't allow for further construction to be cost effective for investors. If further investment were going to happen then the retail market needed to flow back dollars not cents to investors. But life goes on, more fibre strands have been lit, more DWDM systems have been reconditioned to support more lambdas and some more cable systems have been proposed and some have been built. Predictions of the demise of the industry were, of course, exaggerated. But the major flow on from the current generation of DWDM submarine cable systems, as expressed in the impact or pricing at the retail level, to me looks like its been fully realized and unless we see a new means of cramming a couple of orders of magnitude more bits into a long distance submarine fibre pair without spending a couple of orders of magnitude more dollars then we are back to the constant unit price proposition where more bits requires more money. Which is a roundabout way of saying that I'm very sceptical of Tom's exponential optimism in this particular area of infrastructure investment:-) regards, Geoff
Re: Lessons from the AU model
On Jan 21, 2008, at 9:53 PM, Mark Newton wrote: On 22/01/2008, at 10:21 AM, Tom Vest wrote: On Jan 21, 2008, at 6:10 PM, Mark Newton wrote: It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable) So infrastructure spending can (and does) affect the price. Hi Mark, So you're saying that if a cable owner/monopolist simply lit another fiber pair, that would cause them to reduce prices? No, I'm saying that if another infrastructure provider launched itself onto the marketplace, that'd immediately inspire the incumbent who is presently using scarcity as a justification for high prices to light another fiber pair. Actually, what I think you said, this time, is that new capacity investment, when it takes the form of competition, can affect prices. Which goes without saying. But the same thing can happen without new capex, e.g., if a new entrant can purchase (otherwise unutilized) wholesale capacity and multiplex/resell, either at a lower price point or with other value adds, e.g., customer service, etc. It's the competition, not the new investment, that drives the price train... unless of course you outlaw resale or make it technically impossible. It's a bit more complicated than that, but arguing that supply has no effect on price is overly simplistic when there are monopolistic barriers preventing new sources of supply. Put it this way: how would you define (much less calculate) "replacement cost" for an asset whose financing was predicated on a useful of capacity of (x), but which, with fractional additional investment relative to the original outlay, can be leveraged to deliver (x)^4-n capacity -- with n yet to be determined? Must every increment of the now vastly larger resource be priced as it would have been assuming the "original" max cap? How much must the "replacement cost" replace? The original (x) capacity? The as-yet indeterminate (x)^n capacity? The originally anticipated/full scarcity-based/monopoly-backed profits? Whichever one of those comes out to the largest number. But the problem is, you never know which one will come out to the larger number until the proverbial "long run". In some countries, those holding the capacity are betting that (zero to very few sales) + (very high prices) + (freedom from internal and external competition) will deliver the big number. Others are betting that many small slices of a much bigger pie will deliver the big number. Who's to say, today, which one will be right in the end? I bet I can guess which one you'd rather live in however. Eventually the asset will reach its capacity -- we can't keep upgrading things forever. Submarine cable systems also have a useful working life, so even if they haven't reached capacity they'll run out of legs eventually anyway (the ones installed during the dot-com boom are approaching their half-life) When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates. I believe that someone will be able to "build" (i.e., finance) more, when/where more is required. Whether that's the most recent inheritors of the old/distressed assets or someone completely new remains to be seen. So if they don't have a billion or so dollars stored away somewhere, they're selling below replacement value. With very few exceptions there's no "they"; the old "they" is gone, the new "they" didn't take over until fairly recently, didn't bankroll the original construction, and isn't bearing the financing costs of that construction. Once you get acquainted with the power of that ^n, you'll believe ;-) Unfortunately, your location gives you few opportunities to familiarize yourself. Well, no, we have footprint in Australia, the USA and Japan at the moment. We'd have built out to Europe by now if not for the fact that global IP transit is being sold cheaper than transatlantic transmission, so what's the point of building a POP across the pond? Now, given that transatlantic transmission is already artificially cheap due to the acquisition of distressed assets in 2001, what does that tell you about IP transit pricing? It's cheap alright; how much should it be? As for transit < L2 pricing? That tells me that someone has a traffic imbalance problem, and you are the beneficiary; revel in your good fortune :-) Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-) If fine-grained metered pricing comes to the rest of the world, it'll be because people roll over for it (you guys weren't given a choice). No, it won't be because people roll
Re: Lessons from the AU model
Here it stands for Multilateral peering, so in simplistic terms it means you peer with everyone at an exchange. A route server. My issue isn't that they exist (and in fact one network I maintain is a member of three MLPs.) The issue is compelling people to join the MLP. This dissuades large networks from joining in with the public peering game, and is probably harmful to the peering ecology of the region rather than helpful. Given that three of the Gang of Four (Telstra/Reach, Optus/Singtel, AAPT/Telecom NZ) (hey all encumbents!) won't peer in Oz, even with Global Tier1s in Oz, it doesn't seem a big problem as the cost the GoF charge to access their network is so high (Does that mean there are no global tier 1s? :-) In Australia basically almost everyone bar the GoF peers at the major peering fabrics (PIPEx6, WAIXx1, Equinix Sydney) as it gives you such a large chunk of the eyeballs and corporates outside of the GoF. We peer with some of Telstra/Reaches upstreams in the US, but they won't peer in Oz. They won't even peer with some very large global tier2s. Best wishes Andy -- Matthew Moyle-Croft - Internode/Agile - Networks Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: [EMAIL PROTECTED] Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
Re: Lessons from the AU model
On Jan 21, 2008, at 8:20 PM, Roderick Beck wrote: Tom, With all due respect, what are you trying to say in simple, clear English? It is plain wrong to claim the DWDM can explain today's prices on the Atlantic Is it? Sorry, how do you explain it then? An unanticipated/ undiscounted DWDM-driven windfall in salable supply (1), plus competition (2), plus financial distress (3), plus "innovative" transactions (4) are my prime suspects; the fact that every increment of private line capacity sold or otherwise transferred is fungible and can be further multiplexed just accelerates the (2-4) dynamic. I believe that the same thing -- (2) leading to price collapse, leading to (3) -- would have happened even if there been only two TA cables, assuming of course that (2) still applied -- maybe even if there were only one cable with enough upgradeable fiber pairs. Sure, "overinvestment" played some marginal role, but each new facilities platform added a multiple to the available supply, 1x, 2x... 8x... whereas DWDM advances that really kicked in only after gen1 entepreneurial cables were already planned if not in the water, added/ continues to add exponents (1x)^4, (2x)^6... (8x)^n? and that DWDM allows providers to make sufficient ROI.. I didn't say that; I did say that I don't know how to define, much less calculate, "sufficient" ROI. Sufficient for what? When is "sufficient" ever sufficient, assuming you can extract more? If I can build one house, and rent it, manage it, and and maintain it at an overall return that will enable me to subsidize 100% of the purchase of another identical lot and identical materials and labor to build a second house, then maybe I'll decide that that's not really sufficient, e.g., because people demand bigger houses today, or maybe I think I could squeeze harder and get enough for three houses, or maybe four... Which one is "sufficient"? What if I've already got a lot of empty houses, but the market for housing won't deliver prices that satisfy my preferred sufficiency test? I have to believe that the financial authorities (public and private) will recognize these new, previously undiscounted developments, and find a way to integrate this new, previously unrecognized abundance into the broader economy -- e.g., by bankrolling future capacity builds (when there is real demand) even though their builders can't promise and won't deliver bubble-era returns. I have to believe it because I'm not cynical or paranoid enough to buy into any worldview that would lead to other assumptions. I might be proven wrong -- but if I do I'd have to start wondering about all sorted of things, e.g., the 200mpg combustion engine, etc... TV About 70 percent of the operating costs of a TransAtlantic cable are undersea maintenance and undersea maintenance costs are rising (think fuel costs). You can spread those fixed costs over more waves as DWDM improves, but it is not enough in general. I was comparing initial build/launch capex to the incremental capex required for upgrades (granted these costs vary by platform/ technology generation). I'm not surprised that a large/rising share of opex is O&M. The fuel component of O&M isn't incrementing in parallel with additional capacity sales, so why isn't expansion of the de facto sponsorship pool one way to offset the rising costs? TV Sent wirelessly via BlackBerry from T-Mobile. -Original Message- From: Tom Vest <[EMAIL PROTECTED]> Date: Mon, 21 Jan 2008 18:51:24 To:Mark Newton <[EMAIL PROTECTED]> Cc:[EMAIL PROTECTED],[EMAIL PROTECTED],"Geoff Huston" <[EMAIL PROTECTED]>,"Matthew Moyle-Croft" <[EMAIL PROTECTED]>,"Randy Bush" <[EMAIL PROTECTED]>,"Andy Davidson" <[EMAIL PROTECTED]>,"Andrew Odlyzko" <[EMAIL PROTECTED]>,nanog@merit.edu Subject: Re: Lessons from the AU model On Jan 21, 2008, at 6:10 PM, Mark Newton wrote: On 21/01/2008, at 10:49 PM, Tom Vest wrote: In the absence of competition (and esp. in the presence of risk of empowering competitive entrants), supply has no general/necessary effect on prices at all. So excess capacity of a product that is completely monopolized (or priced by cartel fiat, ala OPEC or SC) is largely irrelevant. It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable) So infrastructure spending can (and does) affect the price. Hi Mark, So you're saying that if a cable owner/monopolist simply lit another fiber pair, that would cause them to reduce prices? This mistakes a contingent symptom or tactic or mechanism -- i.e., the int
Re: Lessons from the AU model
On 22/01/2008, at 10:21 AM, Tom Vest wrote: On Jan 21, 2008, at 6:10 PM, Mark Newton wrote: It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable) So infrastructure spending can (and does) affect the price. Hi Mark, So you're saying that if a cable owner/monopolist simply lit another fiber pair, that would cause them to reduce prices? No, I'm saying that if another infrastructure provider launched itself onto the marketplace, that'd immediately inspire the incumbent who is presently using scarcity as a justification for high prices to light another fiber pair. It's a bit more complicated than that, but arguing that supply has no effect on price is overly simplistic when there are monopolistic barriers preventing new sources of supply. Put it this way: how would you define (much less calculate) "replacement cost" for an asset whose financing was predicated on a useful of capacity of (x), but which, with fractional additional investment relative to the original outlay, can be leveraged to deliver (x)^4-n capacity -- with n yet to be determined? Must every increment of the now vastly larger resource be priced as it would have been assuming the "original" max cap? How much must the "replacement cost" replace? The original (x) capacity? The as-yet indeterminate (x)^n capacity? The originally anticipated/full scarcity-based/monopoly-backed profits? Whichever one of those comes out to the largest number. Eventually the asset will reach its capacity -- we can't keep upgrading things forever. Submarine cable systems also have a useful working life, so even if they haven't reached capacity they'll run out of legs eventually anyway (the ones installed during the dot-com boom are approaching their half-life) When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates. So if they don't have a billion or so dollars stored away somewhere, they're selling below replacement value. Once you get acquainted with the power of that ^n, you'll believe ;-) Unfortunately, your location gives you few opportunities to familiarize yourself. Well, no, we have footprint in Australia, the USA and Japan at the moment. We'd have built out to Europe by now if not for the fact that global IP transit is being sold cheaper than transatlantic transmission, so what's the point of building a POP across the pond? Now, given that transatlantic transmission is already artificially cheap due to the acquisition of distressed assets in 2001, what does that tell you about IP transit pricing? Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-) If fine-grained metered pricing comes to the rest of the world, it'll be because people roll over for it (you guys weren't given a choice). No, it won't be because people roll over for it, it'll be because carriers and service providers just get on with it and do it. See my first post to this thread to see the progression which outlines why the introduction of metering by _one_ serious player in any given economy virtually forces every other player to switch to metering as well. Do you think Australian ISPs haven't tried to offer US-style flat-rate services? Of course they have. And they get destroyed in the marketplace. Here's the thing that metering gives you: it stratifies the marketplace. It gives you two classes of customer. One class is customers who know they can live painlessly within the boundaries of whatever quotas you're offering. They don't complain, they just pay their flat monthly bill every month and get on with their lives. The other class is customers who do so much P2P that the imposition of quotas is a painful and unwelcome experience. They whinge and bitch loudly about how awful their ISP is, and migrate en-masse towards whichever ISPs are providing "unlimited" services. The only people who truly care about "unlimited" are the ones who know they can't live within any limits. That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that. That's the dynamic some of you are missing which makes quotas inevitable. If one moderately large player adopts it, the rest of you are going to have to adopt it too. If/when that happens, I'll be lobbying my local gov to turn over the water infrastructure to me so I can replace it with household Evian vending machines; Where I come from household water is already metered, so I'm not sure what you're
Re: Lessons from the AU model
On Jan 21, 2008, at 6:10 PM, Mark Newton wrote:
On 21/01/2008, at 10:49 PM, Tom Vest wrote:
In the absence of competition (and esp. in the presence of risk of
empowering competitive entrants), supply has no general/necessary
effect on prices at all.
So excess capacity of a product that is completely monopolized (or
priced by cartel fiat, ala OPEC or SC) is largely irrelevant.
It goes a bit deeper than that when the monopoly can compound the
problem my artificially constraining capacity by underspending on
infrastructure (e.g., only lighting one pair on a multi-pair cable)
So infrastructure spending can (and does) affect the price.
Hi Mark,
So you're saying that if a cable owner/monopolist simply lit another
fiber pair, that would cause them to reduce prices?
This mistakes a contingent symptom or tactic or mechanism -- i.e.,
the intentionally misleading public explanation ("we're sold out") --
with the real cause/strategy/motives behind "artificially" high
prices...
We get that every day in .au (Transmission on the monopoly route
between Melbourne and Hobart costs 3 times more than transmission
between Sydney and LA; and other potential cable operators have
always known that the monopoly has an excess of supply hidden away
somewhere which they can roll out at bargain basement prices if
a competitor ever arrives in the market)
[ housing ]
Come to think of it, our sector has been struggling with its own
roughly similar terms-of-exchange crisis since about 2004-2005...
arguably driven by very similar prior circumstances as well...
worth investigating a bit further perhaps...
I think the dominant factor that the American internet sector has
been grappling with goes back further than that. It has its origins
in the dot-com boom, when lots of people who didn't have any real
money rolled out enormous infrastructure buildouts. When they
inevitably went broke their infrastructure was bought at cents in
the dollar, enabling the current generation of Internet companies
to behave as if the infrastructure they're using was a lot cheaper
than it really is.
So hardly anyone has been selling below cost, but almost everyone
has been selling below replacement cost.
This makes perfect sense for resources that are not subject to
"multiplexing effects" -- esp. unanticipated (undiscounted)
multiplexing effects.
Put it this way: how would you define (much less calculate)
"replacement cost" for an asset whose financing was predicated on a
useful of capacity of (x), but which, with fractional additional
investment relative to the original outlay, can be leveraged to
deliver (x)^4-n capacity -- with n yet to be determined? Must every
increment of the now vastly larger resource be priced as it would
have been assuming the "original" max cap? How much must the
"replacement cost" replace? The original (x) capacity? The as-yet
indeterminate (x)^n capacity? The originally anticipated/full
scarcity-based/monopoly-backed profits?
So everyone can extract profits for years, making out like bandits
as they grow in to the
excess capacity that was installed between 1999 and 2001, and they
won't have a day of reckoning until they run out of capacity and
find that they haven't been earning enough from their networks to
service the debt they're going to need to take out to perform the
next round of infrastructure upgrades.
Example: You cannot seriously expect me to believe that the price
of transatlantic connectivity actually reflects the cost of laying
cables across the Atlantic. It defies common sense that a Gig-E
tail from NYC to London is priced within an order of magnitude
of a Gig-E tail from NYC to Boston.
Once you get acquainted with the power of that ^n, you'll believe ;-)
Unfortunately, your location gives you few opportunities to
familiarize yourself.
Metered charging systems are, to me, evidence of a realization that
the business model underlying much of the Internet's last five years
is unsustainable. You guys might think they're a novel and
unwelcome arrival at the moment, but give it a few years and we'll
see what happens :-)
If fine-grained metered pricing comes to the rest of the world, it'll
be because people roll over for it (you guys weren't given a choice).
If/when that happens, I'll be lobbying my local gov to turn over the
water infrastructure to me so I can replace it with
household Evian vending machines; and I'd recommend you all get in on
the ground floor in the air market ASAP.
Better be quick though, because the revolution will be just around
the corner...
TV
Re: Lessons from the AU model
On 21/01/2008, at 10:49 PM, Tom Vest wrote: In the absence of competition (and esp. in the presence of risk of empowering competitive entrants), supply has no general/necessary effect on prices at all. So excess capacity of a product that is completely monopolized (or priced by cartel fiat, ala OPEC or SC) is largely irrelevant. It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable) So infrastructure spending can (and does) affect the price. We get that every day in .au (Transmission on the monopoly route between Melbourne and Hobart costs 3 times more than transmission between Sydney and LA; and other potential cable operators have always known that the monopoly has an excess of supply hidden away somewhere which they can roll out at bargain basement prices if a competitor ever arrives in the market) [ housing ] Come to think of it, our sector has been struggling with its own roughly similar terms-of-exchange crisis since about 2004-2005... arguably driven by very similar prior circumstances as well... worth investigating a bit further perhaps... I think the dominant factor that the American internet sector has been grappling with goes back further than that. It has its origins in the dot-com boom, when lots of people who didn't have any real money rolled out enormous infrastructure buildouts. When they inevitably went broke their infrastructure was bought at cents in the dollar, enabling the current generation of Internet companies to behave as if the infrastructure they're using was a lot cheaper than it really is. So hardly anyone has been selling below cost, but almost everyone has been selling below replacement cost. So everyone can extract profits for years, making out like bandits as they grow in to the excess capacity that was installed between 1999 and 2001, and they won't have a day of reckoning until they run out of capacity and find that they haven't been earning enough from their networks to service the debt they're going to need to take out to perform the next round of infrastructure upgrades. Example: You cannot seriously expect me to believe that the price of transatlantic connectivity actually reflects the cost of laying cables across the Atlantic. It defies common sense that a Gig-E tail from NYC to London is priced within an order of magnitude of a Gig-E tail from NYC to Boston. Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-) - mark -- Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Re: Lessons from the AU model
$quoted_author = "Andy Davidson" ; > > I'm really happy for you to sell me some transit as long as I can peer with > you over MLP as well. Small commit. I agree to give you some of my > prefixes over the paid session, but I'm going to put all of my routes and > my customer's routes on the MLP. > > With any luck you wont notice, just be happy that you suddenly seem to be > selling more transit via your upstreams and also peering off more data > too.. :-) Anyone with clue is doing community tagging on learned routes and community filtering on advertised routes and/or as-path filtering on advertised routes. Anyone without clue is welcome to send me their ratecard. :) cheers marty -- "Picture if you will, hundreds of Cisco switches, galloping wildly across the server room, the herd turning as one to avoid the wily sysadmins that are attempting to cull the weak and sick. The slowest switches being captured and herded into racks, where they will spend the rest of their days, passing traffic, dreaming of the freedom they used to have, roaming free in the data center." --That Guy alt.sysadmin.recovery - <[EMAIL PROTECTED]>
Re: Where is the nanog acronym glossary?, was Re: Lessons from the AU model
FWIW.. http://nanog.cluepon.net/index.php/NANOG_Glossary =JeffH
RE: Lessons from the AU model
Is this story relevant? "Undersea cable to slash Aust broadband costs" http://www.nzherald.co.nz/section/2/story.cfm?c_id=2&objectid=10486793 They seem have the sales angle all locked up. Frank -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matthew Moyle-Croft Sent: Sunday, January 20, 2008 8:54 PM To: Geoff Huston Cc: Randy Bush; Andy Davidson; Andrew Odlyzko; nanog@merit.edu Subject: Re: Lessons from the AU model > > Southern Cross cost some US $1B to construct about a decade ago RFS was Nov 2001. They full paid the debt from a US$1.3B cost of construction in Oct 2005. (see http://www.southerncrosscables.com/public/News/newsdetail.cfm?StoryID=14) So, they're making some VERY decent money out of the duopoly with AJC. Hence why Telstra's building their OWN cable to Hawaii. It's cheaper to build than buy! MMC -- Matthew Moyle-Croft - Internode/Agile - Networks Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: [EMAIL PROTECTED] Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
Where is the nanog acronym glossary?, was Re: Lessons from the AU model
And DFZ yields either Duty Free Zone or the ticker symbol for Rg Barry Corp. It took me two weeks to figure out PI (Provider Independent, related to address assignments by a registry.) The FAQ has no mention of a glossary, but it needs to. Dale, AS 26424 >From: Marshall Eubanks <[EMAIL PROTECTED]> >OK, I give and admit my ignorance. What does "MLP" mean in this >context ? > >A google search for "Australia mlp" reveals many hits for "My Little >Pony," >which somehow I doubt is the intended meaning on this list.
Re: Lessons from the AU model
On Mon, Jan 21, 2008, Marshall Eubanks wrote: > > OK, I give and admit my ignorance. What does "MLP" mean in this > context ? > > A google search for "Australia mlp" reveals many hits for "My Little > Pony," > which somehow I doubt is the intended meaning on this list. > > A proper reference would be appreciated. Multi-Lateral Peering vs Bi-Lateral Peering. Adrian
Re: Lessons from the AU model
On 21 Jan 2008, at 14:02, Marshall Eubanks wrote: OK, I give and admit my ignorance. What does "MLP" mean in this context ? A google search for "Australia mlp" reveals many hits for "My Little Pony," which somehow I doubt is the intended meaning on this list. Smile... Here it stands for Multilateral peering, so in simplistic terms it means you peer with everyone at an exchange. A route server. My issue isn't that they exist (and in fact one network I maintain is a member of three MLPs.) The issue is compelling people to join the MLP. This dissuades large networks from joining in with the public peering game, and is probably harmful to the peering ecology of the region rather than helpful. Best wishes Andy
Re: Lessons from the AU model
OK, I give and admit my ignorance. What does "MLP" mean in this context ? A google search for "Australia mlp" reveals many hits for "My Little Pony," which somehow I doubt is the intended meaning on this list. A proper reference would be appreciated. Regards Marshall On Jan 21, 2008, at 8:02 AM, Andy Davidson wrote: On 21 Jan 2008, at 01:43, Martin Barry wrote: $quoted_author = "Andy Davidson" ; .. think about what happens when your customers' routes start appearing through your MLP session as well. Standard practice would be to localpref customer routes over peering routes.[...] The customer's inbound (standard billing metric) won't be affected but their ratio might. I'm really happy for you to sell me some transit as long as I can peer with you over MLP as well. Small commit. I agree to give you some of my prefixes over the paid session, but I'm going to put all of my routes and my customer's routes on the MLP. With any luck you wont notice, just be happy that you suddenly seem to be selling more transit via your upstreams and also peering off more data too.. :-) Andy
Re: Lessons from the AU model
On 21 Jan 2008, at 01:43, Martin Barry wrote: $quoted_author = "Andy Davidson" ; .. think about what happens when your customers' routes start appearing through your MLP session as well. Standard practice would be to localpref customer routes over peering routes.[...] The customer's inbound (standard billing metric) won't be affected but their ratio might. I'm really happy for you to sell me some transit as long as I can peer with you over MLP as well. Small commit. I agree to give you some of my prefixes over the paid session, but I'm going to put all of my routes and my customer's routes on the MLP. With any luck you wont notice, just be happy that you suddenly seem to be selling more transit via your upstreams and also peering off more data too.. :-) Andy
Re: Lessons from the AU model
On Jan 21, 2008, at 6:50 AM, Roderick Beck wrote: I didn't find that very convincing. Overspending on infrastructure does not force prices up or raise long term costs. Excess capacity leads to prices falling in telecom and every other market. Basic economics. Maybe we should go a little more basic still... In the absence of competition (and esp. in the presence of risk of empowering competitive entrants), supply has no general/necessary effect on prices at all. So excess capacity of a product that is completely monopolized (or priced by cartel fiat, ala OPEC or SC) is largely irrelevant. On the other hand, in the specific case of Internet access (retail or wholesale level), prices have a huge impact on demand -- that's called "price elasticity". Given the absence of any mechanism to reveal "natural" prices or "actual" demand, how does one know whether/how much "excess capacity" there is anyway? I'm sure the Americans are experiencing excess capacity right now in housing ... :) American construction / real estate investors and speculators are certainly experiencing excess capacity in the most extreme way, but that's because the fully loaded price of housing has gone up a great deal (and/or, the actual price point is finally being discovered/ revealed) -- at least for the credit-challenged market segment that was the primary engine of demand behind the recent boom. For them especially, lending terms have deteriorated tremendously. Now general interest rates are falling again, but credit requirements are tightening so much that, for many in that group, credit will not be available "at any price". If that seems counterintuitive, think Japan between c. 1993-2005: huge "excess capacity" in real estate, inflation-adjusted interest rates at zero or below, and yet (or perhaps "as a consequence") no one buys, because the banks refuse to lend... Come to think of it, our sector has been struggling with its own roughly similar terms-of-exchange crisis since about 2004-2005... arguably driven by very similar prior circumstances as well... worth investigating a bit further perhaps... TV Sent wirelessly via BlackBerry from T-Mobile. -Original Message- From: Tom Vest <[EMAIL PROTECTED]> Date: Mon, 21 Jan 2008 01:10:16 To:Geoff Huston <[EMAIL PROTECTED]> Cc:Matthew Moyle-Croft <[EMAIL PROTECTED]>, Randy Bush <[EMAIL PROTECTED]>, Andy Davidson <[EMAIL PROTECTED]>, Andrew Odlyzko <[EMAIL PROTECTED]>, nanog@merit.edu Subject: Re: Lessons from the AU model On Jan 20, 2008, at 11:14 PM, Geoff Huston wrote: Matthew Moyle-Croft wrote: Southern Cross cost some US $1B to construct about a decade ago RFS was Nov 2001. They full paid the debt from a US$1.3B cost of construction in Oct 2005. (see http://www.southerncrosscables.com/public/News/newsdetail.cfm? StoryID=14) So, they're making some VERY decent money out of the duopoly with AJC. Yes, that exercise managed to weather the slump in prices a couple of years back when supply far exceeded demand, and then exploit their excellent technical position when demand picked up and translate that position into good revenue streams that appear to be well above initial construction and ongoing operational costs. I don't believe AJC has had a similar story, but others may know more here. Hence why Telstra's building their OWN cable to Hawaii. It's cheaper to build than buy! My comment is that its generally more complicated than that, and from a sufficiently distanced view overspending on infrastructure forces up prices as much as underspending. The only real revenue stream to fund this infrastructure comes from this pool of 24M folk living at the bum end of the Pacific Ocean. Paying for a large number of underutilized cable projects does have a higher total recurrent cost than would be the case of there were efficient sharing of a smaller number of cable projects, and ultimately its consumers who fund this inefficiency in supply. So sometimes competition provides natural incentives for cost efficient investments, that ultimately benefit consumers, and sometimes competition gets it wrong and over-invests because the actors cannot resolve their individual requirements in ways that result in efficient sharing of common venture infrastructure investments, and in such cases the consumer ends up paying for the inefficiency in infrastructure investment. So sometimes it is cheaper to lease than construct, and sometimes its not. For anyone who has the opportunity and inclination to truly monopolize an isolated market, the "best" price to sell critical inputs to a potential competitor is certainly (infinity) -- and if the input is fungible, then *everyone* is a potential competitor. Sure your market grows less/more slow
Re: Lessons from the AU model (was: An Attempt at Economically Rational Pricing: Time Warner Trial)
On Sun, January 20, 2008 11:40 pm, Andy Davidson wrote: > Thanks to the pricing model imposed on last-mile connectivity imposed > by the incumbent, it costs an ISP US$471/Mbit to send data to my > customer[1]. Maybe the same data that's just come all the way from Oz > through my transit for US$10. That would imply that either: - There's an opportunity in the market to provide competing (cheaper) last-mile infrastructure - It really *does* cost that much to provide the last-mile connectivity In actual fact, the last-mile is cheap (at least for the UK, the incumbent is forced to let you have access to the copper at a fairly reasonable price); the business challenge is in balancing the cost of CO equipment and backhaul against revenue from customers served by that CO. Of course, the supplier for facilities at the CO, and quite likely backhaul from it, is the same incumbent; wearing a cynical hat, I might suggest that they are trying their best to make the situation seem more like the second option than the first. Putting your own gear out to the CO has some other wins too - you can control your own contention ratios (and, depending on the CO equipment, sell *different* contention ratios to different customer types), if you have L3 equipment on site then local customer-to-customer traffic doesn't touch your backhaul, and you get to deploy your own choices of technology at your own pace (SDSL, Annex M, EFM, VDSL, etc). Regards, Tim.
Re: Lessons from the AU model
On Sun, 20 Jan 2008, Tom Vest wrote: Let's "hear it" for some of them. Let's "give it" to some of the others. Yep, I was just commenting on the tendency of folks to personalize heroes and villians. There were probably lots of different factors and people involved in both the Australian and Japanese market experiences.
Re: Lessons from the AU model
On Jan 20, 2008, at 11:14 PM, Geoff Huston wrote: Matthew Moyle-Croft wrote: Southern Cross cost some US $1B to construct about a decade ago RFS was Nov 2001. They full paid the debt from a US$1.3B cost of construction in Oct 2005. (see http://www.southerncrosscables.com/public/News/newsdetail.cfm? StoryID=14) So, they're making some VERY decent money out of the duopoly with AJC. Yes, that exercise managed to weather the slump in prices a couple of years back when supply far exceeded demand, and then exploit their excellent technical position when demand picked up and translate that position into good revenue streams that appear to be well above initial construction and ongoing operational costs. I don't believe AJC has had a similar story, but others may know more here. Hence why Telstra's building their OWN cable to Hawaii. It's cheaper to build than buy! My comment is that its generally more complicated than that, and from a sufficiently distanced view overspending on infrastructure forces up prices as much as underspending. The only real revenue stream to fund this infrastructure comes from this pool of 24M folk living at the bum end of the Pacific Ocean. Paying for a large number of underutilized cable projects does have a higher total recurrent cost than would be the case of there were efficient sharing of a smaller number of cable projects, and ultimately its consumers who fund this inefficiency in supply. So sometimes competition provides natural incentives for cost efficient investments, that ultimately benefit consumers, and sometimes competition gets it wrong and over-invests because the actors cannot resolve their individual requirements in ways that result in efficient sharing of common venture infrastructure investments, and in such cases the consumer ends up paying for the inefficiency in infrastructure investment. So sometimes it is cheaper to lease than construct, and sometimes its not. For anyone who has the opportunity and inclination to truly monopolize an isolated market, the "best" price to sell critical inputs to a potential competitor is certainly (infinity) -- and if the input is fungible, then *everyone* is a potential competitor. Sure your market grows less/more slowly, and your near-term revenues are lower -- but you have an untroubled eternity of guaranteed revenue/profit to console you. Building new facilities, even terrifically expensive and completely redundant ones, will cost less than (infinity) -- so of course this makes perfect (if utterly crazy) sense. (and yes, since oceans are not the only features that impose large multipliers on facilities construction costs, this is still a fairly common situation in the world) However, it's not just "consumers" who are paying the price; it's also every other producer in the economy. So if the bottleneck exploiter is not generating as much or more jobs, innovations, private investments, taxes, and other returns to the general economy, compared to the amount that were foregone (never created) by every other productive endeavor in the market, then it's the economy as a whole, present and future, that's paying the price. Here endth the Nanog lesson in economics from me ( :-) ) My only point in entering this thread was to make the observations that the lessons from the AU model may not be very generic - small isolated communities often have a unique set of constraints for investments in communications systems and that often results in different industry structures, different relationships between the actors and often results in different pricing structures in the consumer market. I'm not sure that I'd be confident in generalizing this particular history into anything more generic that would apply to other communities in other parts of the world. (My anti-fatalist rant) This all seems pretty straightforward -- except of course that you, we, and presumably the A-Cs and other fairly observant people can make similar observations, and perhaps even arrive at similar/shared conclusions. Given that, and the potentially extreme/enduring costs (i.e., for everyone but the bottleneck exploiter) borne by everyone who happens to get stuck in the wrong kind of market, shouldn't we look for other, more practical lesson(s) as well? TV
Re: Lessons from the AU model
On Mon, Jan 21, 2008 at 01:20:12PM +1100, Geoff Huston wrote: > > Randy Bush wrote: > > > >and pricing in australia had nothing to do with a monopilist telco with > >a rapacious plan highly well articulated and sold to the govt by an > >arch-capitalist with a silver tongue? > > I don't know about that. However, I do know that relatively small > isolated communities in the bottom end of the South Pacific Ocean have > to make somewhat tough calls in the provisioning of international > connectivity. Satellite is too slow, so it has to be submarine cable. If > you head west on cable then the costs escalate because of the > transcontinental costs just to get the the west cost and the trans- > Indian Ocean runs are either long or run very close to geologically > active areas, and even when you get to Singapore you still have to do a > full trans-Pacific to off load the majority of your traffic, so the end- > to-end delays start to rise. her we start feeling for those poor Oz folk who want but can't get sub 150ms latency.. and who's "buying" that they will forever be held hostage to the low-latency center of the global network, the US. one of the reasons GLORIAD was built was to get asian/eu connectivity below 150ms. > If you head north from the East Coast of > Australia then in theory you can tap into the larger equatorial and > north Pacific east west capacity market, but at the same time the end to > end delays are high and the cost of heading north is almost the same as > the costs of heading north east. And the east west market is highly > uncertain - during business slumps capacity could be had very cheaply, > but when asian demand is strong hen the price escalates very quickly, so > there are some risks with this option. Or you can head directly north > east, as Southern Cross has done some years back. The transmission delay > is as close to optimal as you can get, but it doesn't negate the fact > that at one of the cable is a community of 24M people, which is not > exactly a big market by anyone's metric, and these 24M individuals have > to fuel the entire business case for the infrastructure investment. > > Southern Cross cost some US $1B to construct about a decade ago - I > suspect that a comparable project today would cost somewhere between > $300M and $700M depending on the amount of redundancy you are after, lit > capacity, and the precise landing points of the cable system. But these > days its an investment not without risk, as the existing deployed > systems have a significant capacity overhang in the AU/NZ end of the > market and therefor have the ability to undercut the price of any new > venture if they wished. So new ventures in cable systems in this part of > the world normally requires the buy-in from larger cashed up players. > The consequence is that aspirations of a fiercely competitive market > with follow-on in pricing drops to end consumers tends to be difficult > to realize. I suspect that in these markets it more of a battle between > bankers and investment models than it has any bearing on the technology > or the end user costs in the long run. bankers, investment models, and government commitments tend to drive nearly all infrastructure development. end user costs are rarely, if ever reflected in the costs of infrastructure deployment. --bill
Re: Lessons from the AU model
On Mon, Jan 21, 2008 at 10:46:58AM +1030, Matthew Moyle-Croft wrote: > > The cost is getting out of Oz. Once you get to Japan (Australia Japan > Cable) then it's not that expensive (heck cable station to Tokyo is more > than cable station to USA). > > Currently there are 3 cable systems out of Australia: > > Southern Cross Cable (SXC) - 2 legs to the US, one via Fiji, one via NZ. > Australia Japan Cable (AJC) - one leg from Sydney to Guam and onto Tokyo. > SEAMEWE3 (SMW3) - Perth to Singapore (old, and expensive). > PIPE Network is building Sydney to Guam (PPC-1) which will link up with > a VSNL Cable to Japan and onto the USA. > > There's rumour another cable will be build Perth to Singapore, but it's > been that way for years so let's see. the perth (to anywhere else) cable systems have been planned and evetually binned for years- truth. having been in on the planning and design for a few of them. the curent plan seems to have legs, but is early days. www.ochrenetworks.com - if all goes according to plan, this might go live before the stablization of IPv4... :) --bill > > Matthew Moyle-Croft - Internode/Agile - Networks > Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia > Email: [EMAIL PROTECTED] Web: http://www.on.net > Direct: +61-8-8228-2909 Mobile: +61-419-900-366 > Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 > > "The difficulty lies, not in the new ideas, > but in escaping from the old ones" - John Maynard Keynes
Re: Lessons from the AU model
Matthew Moyle-Croft wrote: Southern Cross cost some US $1B to construct about a decade ago RFS was Nov 2001. They full paid the debt from a US$1.3B cost of construction in Oct 2005. (see http://www.southerncrosscables.com/public/News/newsdetail.cfm?StoryID=14) So, they're making some VERY decent money out of the duopoly with AJC. Yes, that exercise managed to weather the slump in prices a couple of years back when supply far exceeded demand, and then exploit their excellent technical position when demand picked up and translate that position into good revenue streams that appear to be well above initial construction and ongoing operational costs. I don't believe AJC has had a similar story, but others may know more here. Hence why Telstra's building their OWN cable to Hawaii. It's cheaper to build than buy! My comment is that its generally more complicated than that, and from a sufficiently distanced view overspending on infrastructure forces up prices as much as underspending. The only real revenue stream to fund this infrastructure comes from this pool of 24M folk living at the bum end of the Pacific Ocean. Paying for a large number of underutilized cable projects does have a higher total recurrent cost than would be the case of there were efficient sharing of a smaller number of cable projects, and ultimately its consumers who fund this inefficiency in supply. So sometimes competition provides natural incentives for cost efficient investments, that ultimately benefit consumers, and sometimes competition gets it wrong and over-invests because the actors cannot resolve their individual requirements in ways that result in efficient sharing of common venture infrastructure investments, and in such cases the consumer ends up paying for the inefficiency in infrastructure investment. So sometimes it is cheaper to lease than construct, and sometimes its not. Here endth the Nanog lesson in economics from me ( :-) ) My only point in entering this thread was to make the observations that the lessons from the AU model may not be very generic - small isolated communities often have a unique set of constraints for investments in communications systems and that often results in different industry structures, different relationships between the actors and often results in different pricing structures in the consumer market. I'm not sure that I'd be confident in generalizing this particular history into anything more generic that would apply to other communities in other parts of the world.
Re: Lessons from the AU model
On Jan 20, 2008, at 9:36 PM, Sean Donelan wrote: On Mon, 21 Jan 2008, Randy Bush wrote: and pricing in australia had nothing to do with a monopilist telco with a rapacious plan highly well articulated and sold to the govt by an arch-capitalist with a silver tongue? And Japan had the arch-capitalist with a silver tongue, Masayoshi Son, to whack the NTT monopilist telco(s) broadband policy into action. Let's hear it for the arch-capitalists with silver tongues. But the equivalence you seem to be suggesting in fact demonstrates the exact opposite. Which of the A-Cs leveraged (gov-driven) facilities bottleneck relief, and which one has tried to prevent/resist such relief? Anyone care to compare relative price/service differences between AU and JP -- or if that seems too iffy, to compare the magnitude of within-country change in service price/quality over the last 5-6 years? Let's "hear it" for some of them. Let's "give it" to some of the others. TV
Re: Lessons from the AU model
On Mon, Jan 21, 2008, Mark Newton wrote: > We don't really have a lot of ratio-limit issues over here. Nobody > is going to say, "Our traffic is way imbalanced so I'm not peering > with you anymore," when transit costs hundreds of bucks per megabit. .. its starting. Adrian
Re: Lessons from the AU model
Southern Cross cost some US $1B to construct about a decade ago RFS was Nov 2001. They full paid the debt from a US$1.3B cost of construction in Oct 2005. (see http://www.southerncrosscables.com/public/News/newsdetail.cfm?StoryID=14) So, they're making some VERY decent money out of the duopoly with AJC. Hence why Telstra's building their OWN cable to Hawaii. It's cheaper to build than buy! MMC -- Matthew Moyle-Croft - Internode/Agile - Networks Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: [EMAIL PROTECTED] Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
Re: Lessons from the AU model
On 21/01/2008, at 12:43 PM, Martin Barry wrote:
This was basically setting up the next comment which was in relation
to how
this situation ("my customer is now at a multi-lateral peering point
I'm on")
is not really an issue as far as the bean-counters are concerned.
Unless any
ratio limit you have was applicable to that customer.
We don't really have a lot of ratio-limit issues over here. Nobody
is going to say, "Our traffic is way imbalanced so I'm not peering
with you anymore," when transit costs hundreds of bucks per megabit.
The industry in .au is very firmly stratified into one group
containing Telstra, Singtel/Optus, AAPT/TNZ and MCI-703, and another
group containing everyone else. The everyone-else crowd (which
makes up well over half of the domestic marketplace) is perfectly
happy to exchange traffic with each other on almost any terms
whatsoever if it means they can reduce or elminiate their financial
commitments to the other four.
- mark
--
Mark Newton Email: [EMAIL PROTECTED]
(W)
Network Engineer Email:
[EMAIL PROTECTED] (H)
Internode Systems Pty Ltd Desk: +61-8-82282999
"Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Re: Lessons from the AU model
On Mon, 21 Jan 2008, Randy Bush wrote: and pricing in australia had nothing to do with a monopilist telco with a rapacious plan highly well articulated and sold to the govt by an arch-capitalist with a silver tongue? And Japan had the arch-capitalist with a silver tongue, Masayoshi Son, to whack the NTT monopilist telco(s) broadband policy into action. Let's hear it for the arch-capitalists with silver tongues.
Re: Lessons from the AU model
Geoff Huston wrote: Randy Bush wrote: and pricing in australia had nothing to do with a monopilist telco with a rapacious plan highly well articulated and sold to the govt by an arch-capitalist with a silver tongue? I don't know about that. However, I do know that relatively small isolated communities in the bottom end of the South Pacific Ocean have to make somewhat tough calls in the provisioning of international connectivity. Satellite is too slow, so it has to be submarine cable. If you head west on cable then the costs escalate because of the transcontinental costs just to get the the west cost and the trans- Indian Ocean runs are either long or run very close to geologically active areas, and even when you get to Singapore you still have to do a full trans-Pacific to off load the majority of your traffic, so the end- to-end delays start to rise. If you head north from the East Coast of Australia then in theory you can tap into the larger equatorial and north Pacific east west capacity market, but at the same time the end to end delays are high and the cost of heading north is almost the same as the costs of heading north east. And the east west market is highly uncertain - during business slumps capacity could be had very cheaply, but when asian demand is strong hen the price escalates very quickly, so there are some risks with this option. Or you can head directly north east, as Southern Cross has done some years back. The transmission delay is as close to optimal as you can get, but it doesn't negate the fact that at one of the cable is a community of 24M people, which is not exactly a big market by anyone's metric, and these 24M individuals have to fuel the entire business case for the infrastructure investment. Southern Cross cost some US $1B to construct about a decade ago - I suspect that a comparable project today would cost somewhere between $300M and $700M depending on the amount of redundancy you are after, lit capacity, and the precise landing points of the cable system. But these days its an investment not without risk, as the existing deployed systems have a significant capacity overhang in the AU/NZ end of the market and therefor have the ability to undercut the price of any new venture if they wished. So new ventures in cable systems in this part of the world normally requires the buy-in from larger cashed up players. The consequence is that aspirations of a fiercely competitive market with follow-on in pricing drops to end consumers tends to be difficult to realize. I suspect that in these markets it more of a battle between bankers and investment models than it has any bearing on the technology or the end user costs in the long run. QED!
Re: Lessons from the AU model
Randy Bush wrote: and pricing in australia had nothing to do with a monopilist telco with a rapacious plan highly well articulated and sold to the govt by an arch-capitalist with a silver tongue? I don't know about that. However, I do know that relatively small isolated communities in the bottom end of the South Pacific Ocean have to make somewhat tough calls in the provisioning of international connectivity. Satellite is too slow, so it has to be submarine cable. If you head west on cable then the costs escalate because of the transcontinental costs just to get the the west cost and the trans- Indian Ocean runs are either long or run very close to geologically active areas, and even when you get to Singapore you still have to do a full trans-Pacific to off load the majority of your traffic, so the end- to-end delays start to rise. If you head north from the East Coast of Australia then in theory you can tap into the larger equatorial and north Pacific east west capacity market, but at the same time the end to end delays are high and the cost of heading north is almost the same as the costs of heading north east. And the east west market is highly uncertain - during business slumps capacity could be had very cheaply, but when asian demand is strong hen the price escalates very quickly, so there are some risks with this option. Or you can head directly north east, as Southern Cross has done some years back. The transmission delay is as close to optimal as you can get, but it doesn't negate the fact that at one of the cable is a community of 24M people, which is not exactly a big market by anyone's metric, and these 24M individuals have to fuel the entire business case for the infrastructure investment. Southern Cross cost some US $1B to construct about a decade ago - I suspect that a comparable project today would cost somewhere between $300M and $700M depending on the amount of redundancy you are after, lit capacity, and the precise landing points of the cable system. But these days its an investment not without risk, as the existing deployed systems have a significant capacity overhang in the AU/NZ end of the market and therefor have the ability to undercut the price of any new venture if they wished. So new ventures in cable systems in this part of the world normally requires the buy-in from larger cashed up players. The consequence is that aspirations of a fiercely competitive market with follow-on in pricing drops to end consumers tends to be difficult to realize. I suspect that in these markets it more of a battle between bankers and investment models than it has any bearing on the technology or the end user costs in the long run.
Re: Lessons from the AU model
$quoted_author = "Randy Bush" ;
>
>> Likely to result in assymetric routing as the customer prefers peering
>> routes over transit.
>
> omg! asymmetric routing on the internet! world at eleven, end of news
> predicted! :)
:)
This was basically setting up the next comment which was in relation to how
this situation ("my customer is now at a multi-lateral peering point I'm on")
is not really an issue as far as the bean-counters are concerned. Unless any
ratio limit you have was applicable to that customer.
cheers
marty
--
with usenet gone, we just don't teach our kids entertainment-level hyperbole
any more. --Paul Vixie
http://www.merit.edu/mail.archives/nanog/2006-01/msg00593.html
Re: Lessons from the AU model
On Mon, Jan 21, 2008, Andy Davidson wrote: > >Peering in Oz is MLPA. This leads to no one worrying about having > >to be found to form peering relationships, so peeringdb is > >incomplete at best. I've tried to encourage people to add their > >data in. > > Is it always compulsory ? (I just did some legwork and read the WAIX > policies, and it seems to be mandatory here) This surprises me, > Multi-lateral peering is great for lots of networks, but really bad > for others, and (if forced) probably acts as a barrier to the bigger > networks from taking part in any public peering Early on, the large providers still wouldn't peer. These MLPA IX'es were formed primarily for small providers to dodge >$2000 a megabit a month transit costs. These days, the small-now-large providers MLPA'ing at the local IX are starting to discover why MLPA is great for little players but financially silly for larger ones. That said, peering at a state IX doesn't preclude you from peering with one of the telco's (as far as I gather - I've been out of this industry for quite a while!) and for the sake of the growth of IP in Australia I'd like to see the bulk of peering still be MLPA. Cracking a bilateral peering nut in Australia would be .. funny to watch. > >1/3 from (expensive) transit to the "Gang of Four) who won't peer > > and acts as an incentive to pull out of the agreement as networks > grow .. think about what happens when your customers' routes start > appearing through your MLP session as well. Then you make absolutely sure you only announce your local routes to each local MLP IX. If you announce your entire network to it then you should know what you're doing. :) A few WAIX participants do that as the cost of hauling it over their WAN links between capital cities is smaller than farming it off via transit (I'm guessing.) > I can think of some MLP-only exchanges in Europe, but I can't think of > any that do significant traffic. Completely different scale of things :) And you'll find people who run (mostly) open peering policies there too. Adrian
Re: Lessons from the AU model
Standard practice would be to localpref customer routes over peering routes. unless unusual agreements exist with peers, this is pretty much normal config everywhere ever since vaf whacked asp and me in '96. otherwise, if you peer multiple places, the peer sees inconsistent routes, which usually gets you a nastygram from rigorous peers. Likely to result in assymetric routing as the customer prefers peering routes over transit. omg! asymmetric routing on the internet! world at eleven, end of news predicted! :) meeting at multiple points and hot potato gets you asymmetry right off the bat. randy
Re: Lessons from the AU model
$quoted_author = "Andy Davidson" ; > > .. think about what happens when your customers' routes start appearing > through your MLP session as well. Standard practice would be to localpref customer routes over peering routes. Likely to result in assymetric routing as the customer prefers peering routes over transit. The customer's inbound (standard billing metric) won't be affected but their ratio might. BTDTGTTS cheers marty -- "It'd go along with the 'Caution - hidden dip' signs on some of then more undulating local roads, which always have me looking out for concealed bowls of guacamole." --Tanuki alt.sysadmin.recovery - <[EMAIL PROTECTED]>
Re: Lessons from the AU model
Andy Davidson wrote: On 21 Jan 2008, at 00:16, Matthew Moyle-Croft wrote: Andy Davidson wrote: - Am I peering widely enough ? Should I actually be stuffing a switch under the floor in my employer's suite and letting my buddies plug in ? Peeringdb knows about eight exchanges in a developed economy of 20 million people. We have more than eight in single cities of Europe. Peering in Oz is MPLA. This leads to no one worrying about having to be found to form peering relationships, so peeringdb is incomplete at best. I've tried to encourage people to add their data in. Is it always compulsory ? Yes. (I just did some legwork and read the WAIX policies, and it seems to be mandatory here) This surprises me, Multi-lateral peering is great for lots of networks, but really bad for others, and (if forced) probably acts as a barrier to the bigger networks from taking part in any public peering Well, there's basically only 5 networks of any size that don't participate (Telstra, Optus Singtel, AAPT TNZ, MCI and SPT (AS9942) who aren't that big). So, clearly it's not a big issue. What's interesting is the kinds of people that DO participate (Asia Netcom, VSNL (Oz only) etc). 1/3 from (expensive) transit to the "Gang of Four) who won't peer and acts as an incentive to pull out of the agreement as networks grow .. think about what happens when your customers' routes start appearing through your MLP session as well. It's an issue. But overall it's not big enough of one. We all gain a lot more by peering together than not. I can think of some MLP-only exchanges in Europe, but I can't think of any that do significant traffic. Yeah - I can understand why MLP is not big elsewhere (we peer in the US so I do the peering there - it's very different, but also the traffic levels are very different. We all share a lot of content here and so the savings amongst 7 of the 10 top broadband companies in Oz make it all worth while. MLP is what works here - the GoF believe that their transit is special and won't peer with anyone else in Oz. It's partially a government mistake from the late 90s. MMC Andy -- Matthew Moyle-Croft - Internode/Agile - Networks Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: [EMAIL PROTECTED] Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
Re: Lessons from the AU model
On 21 Jan 2008, at 00:16, Matthew Moyle-Croft wrote: Andy Davidson wrote: - Am I peering widely enough ? Should I actually be stuffing a switch under the floor in my employer's suite and letting my buddies plug in ? Peeringdb knows about eight exchanges in a developed economy of 20 million people. We have more than eight in single cities of Europe. Peering in Oz is MPLA. This leads to no one worrying about having to be found to form peering relationships, so peeringdb is incomplete at best. I've tried to encourage people to add their data in. Is it always compulsory ? (I just did some legwork and read the WAIX policies, and it seems to be mandatory here) This surprises me, Multi-lateral peering is great for lots of networks, but really bad for others, and (if forced) probably acts as a barrier to the bigger networks from taking part in any public peering 1/3 from (expensive) transit to the "Gang of Four) who won't peer and acts as an incentive to pull out of the agreement as networks grow .. think about what happens when your customers' routes start appearing through your MLP session as well. I can think of some MLP-only exchanges in Europe, but I can't think of any that do significant traffic. Andy
Re: Lessons from the AU model
On 21/01/2008, at 11:02 AM, Randy Bush wrote: and pricing in australia had nothing to do with a monopilist telco with a rapacious plan highly well articulated and sold to the govt by an arch-capitalist with a silver tongue? It used to, but not so much now. The access tail price is still dominated by Telstra's macinations, but national SDH capacity is relatively competitive these days, and Telstra are almost completely uninvolved in transpacific pricing. Transpacific pricing is more driven by the duopoly effects of AJC and Southern Cross. Will be interesting to see how PPC-1 shakes the market up next year. We're looking forward to it. - mark -- Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Re: Lessons from the AU model
and pricing in australia had nothing to do with a monopilist telco with a rapacious plan highly well articulated and sold to the govt by an arch-capitalist with a silver tongue? randy
Re: Lessons from the AU model
- What's my traffic to south Asia and the other apnic regions ? Can I save some money buying *partial* routes from a large player in this region. Or is the problem that actually it's the transport to *anywhere* out of Australia ? The cost is getting out of Oz. Once you get to Japan (Australia Japan Cable) then it's not that expensive (heck cable station to Tokyo is more than cable station to USA). Currently there are 3 cable systems out of Australia: Southern Cross Cable (SXC) - 2 legs to the US, one via Fiji, one via NZ. Australia Japan Cable (AJC) - one leg from Sydney to Guam and onto Tokyo. SEAMEWE3 (SMW3) - Perth to Singapore (old, and expensive). PIPE Network is building Sydney to Guam (PPC-1) which will link up with a VSNL Cable to Japan and onto the USA. There's rumour another cable will be build Perth to Singapore, but it's been that way for years so let's see. All of these cable systems are charging high rates compared to what you guys are used to across the Atlantic for instance (order of magnitude or more). - Am I peering widely enough ? Should I actually be stuffing a switch under the floor in my employer's suite and letting my buddies plug in ? Peeringdb knows about eight exchanges in a developed economy of 20 million people. We have more than eight in single cities of Europe. Peering in Oz is MPLA. This leads to no one worrying about having to be found to form peering relationships, so peeringdb is incomplete at best. I've tried to encourage people to add their data in. However, domestic peering here is alive and well. We get 2/3 of our domestic capacity from peering. 1/3 from (expensive) transit to the "Gang of Four) who won't peer (Telstra/Optus Singtel/AAPT Telecom NZ/MCI 703). (AS703 will peer sometimes). Unfortunately the 1/3 transit for domestic is outrageously expensive too. (Well over US$120/mbps in large quantity). -- Matthew Moyle-Croft - Internode/Agile - Networks Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: [EMAIL PROTECTED] Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
Re: Lessons from the AU model (was: An Attempt at Economically Rational Pricing: Time Warner Trial)
Hi, On 20 Jan 2008, at 16:37, Andrew Odlyzko wrote: The more sensible end of town pays about $80 per month for about 40 Gbytes of quota, give or take, depending on the ISP. After that they get shaped to 64 kbps unless they want to pay more for more quota. I replied offlist to Andrew with some ideas, but I have been thinking more about the econometric model of Australian connectivity, and how interesting it is. When transit is costing $250 per megabit per month, there aren't many other options. [...] On Sun Jan 20, Matthew Moyle-Croft wrote: Having a cap and slowing down afterward (64kbps or 128kbps are typical) is what worked here in Oz. The grass is always greener of course, but when I think about why ISPs in the UK have to cap fairly aggressively (bear in mind I pay the figure Andrew cited is typical for internet access in AU, and have a smaller quota !), there are aspects of the Australian problem that I am envious of. Australia has a relatively small population (c. 20m) which would act as a small ceiling for demand, but the vast majority of the population live in relatively close proximity. Density is highest in coastal regions which makes it ideal for fiber landing ! I will trust Andrew's numbers of $80 for a 40GB cap. The UK has a population of c. 60m, and population density is high (12k people per square mile in London). I'm more likely to pay less than $10/Mbit for global transit. The small country and concentration of POPs in key metropolitan areas makes interconnection cheap. Exchange membership and participation is hugely popular (>600 networks peer in London, 415 peer exclusively in the UK). And yet I pay US$70 to my ISP for residential connectivity and my cap is 30GB. Why is this ? Thanks to the pricing model imposed on last-mile connectivity imposed by the incumbent, it costs an ISP US$471/Mbit to send data to my customer[1]. Maybe the same data that's just come all the way from Oz through my transit for US$10. It used to be the case that global transit was very expensive wherever deployed in the world. As pricing fell, this fueled innovation and created demand for connectivity at every level - domestic, data centre, enterprise, carrier ... The price of connectivity to Australia is likely to fall, because as the opportunity to sell connectivity increases, so should the number of fibers running to Australia (if we were all in the same room, this is the point that Rod would wave his arms, leap out of his seat, and announce that he's already half the way there and would get there first, so noone else needs to try :-) ). And as economies way off to the west of the country grow and stabilise, then the options for sea routes to the country will grow. In the mean time, that $250 figure is a market price. Attempt to modify the market conditions will change the price. But perhaps there are commercial activities that could stimulate demand for consumer IP services - here's some ideas for thought - What's my traffic to south Asia and the other apnic regions ? Can I save some money buying *partial* routes from a large player in this region. Or is the problem that actually it's the transport to *anywhere* out of Australia ? - Am I peering widely enough ? Should I actually be stuffing a switch under the floor in my employer's suite and letting my buddies plug in ? Peeringdb knows about eight exchanges in a developed economy of 20 million people. We have more than eight in single cities of Europe. - So transit pricing sucks. But that's one of my costs as an operator. What's the pricing of a footprint in carrier hotels ? Are there enough carrier hotels ? How much am I paying for power ? If real estate and power is cheap in AU, then shouldn't content network operators in places where power and space is expensive already be planning how to pop up in Australia ? - What about local content ? Why is so much traffic leaving the country ? Does someone need to be extremely plucky and offer bargain basement content hosting in the continent ? If AU entrepreneurs are ignoring the online channel for retail and entertainment, then who wants to jump on a plane and turn this situation around with me ? :-) - OK, how about we proxy certain types of content unless our users opt out. Any cache hits don't contribute to their 40GB monthly download. If transit is the problem, then offer financial incentives to your users to help you not pay for it. If you're peering IP, why not start "peering" your top cache hits between providers too ? - P2P is probably a problem for AU networks. Contrary to most policy makers, there are services which use p2p as a transport, that don't involve the distribution of copyright material without consent. The next generation services that use p2p as a transport, e.g. Joost, have said to be trying to build proximity aw
