Re: Measured Internet good v. bad traffic
Oops, didn't fully understand the post before I hit reply. Ignore that little rant.
RE: Measured Internet good v. bad traffic
I realize that you rescinded this post, but I still think it's worth responding to the arguments to show why they're wrong. On Sun, Aug 31, 2003 at 03:44:00PM -0700, David Schwartz wrote: If you don't want to, don't accept that traffic. It's just like a store stocking Christmas toys. If they don't sell, you're stuck with them. A customer will only pay for what he wants, not what you think he should want. My car gets horrible mileage, therefore, I will only pay for the amount of gas that SHOULD be used according to the factory sticker, not the rest burned up by my fuel-inefficient driving methods. Suppose most people did get the posted gas mileage, but one or two people suddenly got stuck with a bill for twenty times the usual amount. It would be very reasonable for car companies to 'insure' people against being that unlucky person because people do try to budget for fuel. Unlike DoS attacks, however, this hits everyone evenly anyway. It isn't a large, unpredictable cost over which the customer has no control. I just rented a truck. A construction detour forced me to put more mileage on the truck than I intended, therefore, I will only pay for the mileage that I would have accumulated had there been no detours due to construction. Some rental companies actually do this. They bill you based upon the expected mileage for a trip (usually subject to some limit to discourage lying). If people really did fear this (if it was significant), they might well seek insurance against such unexpected expenses and it would make sense for the rental agencies to provide this insurance themselves. Another key difference is that there's nothing truck rental agencies can do about construction. On the other hand, there are many things ISPs can do about DoS attacks. No, this is not a store stocking Christmas toys, or a Progressive(tm) insurance commercial. This is bandwidth. Right, and it's a product just like any other product that can be sold by widely differing business models. Make sure you and your customer (or you and your ISP) have a common understanding. Any fixed rate contract has some insurance aspects. All of these arguments reflect technical thinking rather than business thinking. The business model that seems obvious to you is not the only possible business model. What seems reasonable from one side of the table seems reasonable from the other. Again, I present the factual counter-exemple. I have never had a problem getting an ISP to agree not to bill for DoS attacks provided notification was timely (and I have negotiated on others' behalf several times). Some did insist on a reasonable per-incident fee ($400-$500), though oddly none have ever actually charged for that fee. By the way, another thing I always negotiate for is the ability to opt-out of any permanent filtering of apparently valid traffic. We, of course, allow things like spoof prevention and emergency filters to deal with worms or other problems. DS
Re: Measured Internet good v. bad traffic
snip I sympathize with the customer. There is no reason he should pay for traffic he did not request and does not want. If unwanted traffic raises your cost of providing the service for which you are paid (providing wanted traffic) then you should raise your rates. snip Then why should _I_ bear the cost of traffic destined to you? Somebody has to pay, and I'ld rather you pay for it, you seem to believe that I (and all of the rest of PROVIDER's customers should pay). Which is more or less fair?
RE: Measured Internet good v. bad traffic
snip I sympathize with the customer. There is no reason he should pay for traffic he did not request and does not want. If unwanted traffic raises your cost of providing the service for which you are paid (providing wanted traffic) then you should raise your rates. snip Then why should _I_ bear the cost of traffic destined to you? If you don't want to, don't accept that traffic. It's just like a store stocking Christmas toys. If they don't sell, you're stuck with them. A customer will only pay for what he wants, not what you think he should want. Somebody has to pay, and I'd rather you pay for it, you seem to believe that I (and all of the rest of PROVIDER's customers should pay). Of course the customer pays for it however you slice it. Which is more or less fair? Both are equally fair if all sides explicitly agree. Burger King could, for example, raise prices in high crime areas, that would be perfectly fair since the crime costs them. But they could also decide that customers prefer more uniformity in pricing and feel they should not pay for other people's crimes, so they'll distribute the cost of crime by raising prices for everyone. Similary, customers don't want to worry about DoS attacks over which they have no control. They may not feel it's fair to pay for something they do not want. So many ISPs find that the uniformity of pricing is worth more to their customers. Neither is inherently more fair or more unfair. They're just different approaches. My point is not that it's unfair to make customers pay for DoS attack traffic. My point is that one-sided arguments make no actual business sense. There is no 'unfair' when all participants agree. The one-sided views are harmful because the people who hold them may be totally blind-sided when their customers come back with the other side, a side they never really looked at because it seemed unreasonable at first blush. Yes, businesses routinely eat costs that affect transactions non-uniformly and build them into more uniform prices. They do this because it provides better billing predictability to their customers. A customer's understand of your traffic may not be the same as your understanding and you had better make sure you make it clear. If FedEx delivers a bomb to me postage due, they had better not expect me to pay the charges. I don't want it and the fact that someone told FedEx I wanted it doesn't change anything. DS
Re: Measured Internet good v. bad traffic
On Sun, Aug 31, 2003 at 03:44:00PM -0700, David Schwartz wrote: If you don't want to, don't accept that traffic. It's just like a store stocking Christmas toys. If they don't sell, you're stuck with them. A customer will only pay for what he wants, not what you think he should want. My car gets horrible mileage, therefore, I will only pay for the amount of gas that SHOULD be used according to the factory sticker, not the rest burned up by my fuel-inefficient driving methods. I just rented a truck. A construction detour forced me to put more mileage on the truck than I intended, therefore, I will only pay for the mileage that I would have accumulated had there been no detours due to construction. No, this is not a store stocking Christmas toys, or a Progressive(tm) insurance commercial. This is bandwidth.
RE: Measured Internet good v. bad traffic
At 02:45 AM 8/28/2003, David Schwartz wrote: No that wouldnt work, that was be an analogy to non-usage based eg I buy a 10Mb port from you and you dont charge me extra for unwanted bandwidth across your network.. The point is that 'usage' is supposed to be 'what you use', not what somebody else uses. 'My' traffic is the traffic I want, not the traffic you try to give me that I don't want. An Internet-connected line is like an 800 phone line. You get connected, you advertise your presence, you have no control over who calls, you pay the bill for the incoming calls. That's just *how it is*. jc
RE: Measured Internet good v. bad traffic
At 02:45 AM 8/28/2003, David Schwartz wrote: No that wouldnt work, that was be an analogy to non-usage based eg I buy a 10Mb port from you and you dont charge me extra for unwanted bandwidth across your network.. The point is that 'usage' is supposed to be 'what you use', not what somebody else uses. 'My' traffic is the traffic I want, not the traffic you try to give me that I don't want. An Internet-connected line is like an 800 phone line. You get connected, you advertise your presence, you have no control over who calls, you pay the bill for the incoming calls. That's just *how it is*. jc The last time I went looking for more bandwidth from a new provider (5 months ago or so), I talked to five major providers. I told each one that we would not pay for attack traffic after we notified them of the problem but were willing to pay a reasonable 'per-incident' fee (say $500). Not one of these providers had any problem with that. So it's not how it is. DS
RE: Measured Internet good v. bad traffic
On Wed, 27 Aug 2003, David Schwartz wrote: I mean if the traffic were unrealistically to increase so that bad traffic was 50% of all traffic we would all have to double our circuit and router capacity and you either pass that cost on directly (charge for extra usage) or indirectly (increase the $ per Mb) to the user. I think you're right to say that if thats not acceptable to the user then usage based billing should be avoided for them but ultimately they will still incur the cost as you increase prices over time to foot the cost of increasing overheads. Analogically, imagine if Burger King kept getting shipments of buns that they didn't want but still had to pay for. Their customers would get pretty pissed if BK added an 'unwanted bun' charge to their bill (absent specific prior agreement). I pay for the food I order, not the food BK's suppliers ship to BK. Of course, it's reasonable for BK to raise their prices for the costs of having to deal with the unwanted food. No that wouldnt work, that was be an analogy to non-usage based eg I buy a 10Mb port from you and you dont charge me extra for unwanted bandwidth across your network.. I sympathize with the customer. There is no reason he should pay for traffic he did not request and does not want. If unwanted traffic raises your cost of providing the service for which you are paid (providing wanted traffic) then you should raise your rates. Thats the nature of the Internet which is what you're buying.. you get a permanent supply of unwanted packets, attacks, spam, viruses etc. If you want to avoid it dont connect to the Internet. In principle, one could certainly enter into an agreement where the customer agrees to bear the costs of unwanted traffic in exchange for a lower rate. But I certainly wouldn't assume the customer agreed to pay for traffic he doesn't want and didn't ask for unless the contract explicitly says so. Most contracts define traffic as the averaged rate across the interface, they dont look into what that traffic is and whether anyone requested it. In this sense the comparisons between internet traffic and toll phone calls breaks down, its also the basis for an argument on settlement free bilateral peering ;p And for those people entering into contracts, make sure the contract is clear about what happens with DoS attacks and where the billable traffic is measured. Otherwise you might be pretty surprised if you get a bill for 250Mbps of traffic when you contracted for a 45Mbps circuit. Indeed, but most contracts are either 95 percentile or another kind of smoothed average.. if however it specifies for example you are charged on the peak 5 minute average in the month you could be in trouble! For those dealing with contracts already in place, if your provider argues that you are responsible for all attack traffic no matter what, ask them if that means you could possibly get billed for 1Gbps of traffic even though you only bought a T1. Presumably as the measurement is on the rate across the interface this couldnt happen.. Steve
RE: Measured Internet good v. bad traffic
On Wed, 27 Aug 2003, David Schwartz wrote: Analogically, imagine if Burger King kept getting shipments of buns that they didn't want but still had to pay for. Their customers would get pretty pissed if BK added an 'unwanted bun' charge to their bill (absent specific prior agreement). I pay for the food I order, not the food BK's suppliers ship to BK. Of course, it's reasonable for BK to raise their prices for the costs of having to deal with the unwanted food. No that wouldnt work, that was be an analogy to non-usage based eg I buy a 10Mb port from you and you dont charge me extra for unwanted bandwidth across your network.. The point is that 'usage' is supposed to be 'what you use', not what somebody else uses. 'My' traffic is the traffic I want, not the traffic you try to give me that I don't want. I sympathize with the customer. There is no reason he should pay for traffic he did not request and does not want. If unwanted traffic raises your cost of providing the service for which you are paid (providing wanted traffic) then you should raise your rates. Thats the nature of the Internet which is what you're buying.. you get a permanent supply of unwanted packets, attacks, spam, viruses etc. If you want to avoid it dont connect to the Internet. I don't want to avoid it, I just don't want to be charged for what I do not want. If someone FedExed me a bomb postage due, there are many things FedEx might do, but to try to get me to pay the postage is not one of them. There are few things I can do to stop FedEx from delivering me a bomb and there are many things FedEx can do to stop them from delivering one to me. In general, the customer cannot fix the problem. In principle, one could certainly enter into an agreement where the customer agrees to bear the costs of unwanted traffic in exchange for a lower rate. But I certainly wouldn't assume the customer agreed to pay for traffic he doesn't want and didn't ask for unless the contract explicitly says so. Most contracts define traffic as the averaged rate across the interface, they dont look into what that traffic is and whether anyone requested it. In this sense the comparisons between internet traffic and toll phone calls breaks down, its also the basis for an argument on settlement free bilateral peering ;p Suppose, for example, my provider's network management scheme pings my end of the link every once in a while to see if the link is up. Suppose further this ping made a dent in my bill, so the provider decides to ping more often, say five times a second with large packets to be *sure* the link is reliable. Do you seriously think it's reasonable for me to pay for this traffic? And for those people entering into contracts, make sure the contract is clear about what happens with DoS attacks and where the billable traffic is measured. Otherwise you might be pretty surprised if you get a bill for 250Mbps of traffic when you contracted for a 45Mbps circuit. Indeed, but most contracts are either 95 percentile or another kind of smoothed average.. if however it specifies for example you are charged on the peak 5 minute average in the month you could be in trouble! There is no limit to how long a DoS attack can last. And your provider has no incentive to trace/filter if he gets a major profit if he can just make that attack last a few more hours. Even with 95 percentile billing, seven hours of 100Mbps can push your 95% from 5Mbps up to 12Mbps very easily. Heck, stalling from 6PM when the attack starts until 10AM the next morning could make them a bundle. For those dealing with contracts already in place, if your provider argues that you are responsible for all attack traffic no matter what, ask them if that means you could possibly get billed for 1Gbps of traffic even though you only bought a T1. Presumably as the measurement is on the rate across the interface this couldnt happen.. If the contract isn't explicit, it costs the provider just as much to drop the traffic at the interface as it does to send it over the interface. So the 'we have to pay for it' argument is not limited to the interface rate. By definition, anything two parties agree to with full knowledge is fair to both of them. How DoS attacks are handled should be part of the negotiation of any ISP/customer agreement. However, for many of the contracts I've seen the contract was silent and ambiguous. For a 95 percentile agreement, it's reasonable for the customer to take responsibility for DoS traffic until he makes a request to the provider's NOC. It's also reasonable for the provider to charge a fixed 'incident fee' for each attack that requires NOC and network resources. It is not reasonable for the incentive structure to reward the NOC for doing nothing and penalize them for any attempt to help.
RE: Measured Internet good v. bad traffic
On Thu, 28 Aug 2003, David Schwartz wrote: The point is that 'usage' is supposed to be 'what you use', not what somebody else uses. 'My' traffic is the traffic I want, not the traffic you try to give me that I don't want. Okay but in Internet terms the receiver usually pays for the traffic without necessarily initiating it, this is different from everyday experience of FedEx-ing a parcel or making a telephone call in which it is the sender who picks up the charge. This isnt really a quesion its more a statement of fact.. I don't want to avoid it, I just don't want to be charged for what I do not want. Which is a natural enough reaction but you dont necessarily get what you want :) I cant see any ISP negotiating a transit contract which takes account of unwanted traffic, apart from the fact that there is a real cost which has to be borne somewhere (I previously suggested if they didnt charge you the Mbs they would just increase the $$$s to compensate) its just too complicated from a billing point of view to work this out. Suppose, for example, my provider's network management scheme pings my end of the link every once in a while to see if the link is up. Suppose further this ping made a dent in my bill, so the provider decides to ping more often, say five times a second with large packets to be *sure* the link is reliable. Do you seriously think it's reasonable for me to pay for this traffic? That would be deliberate on the providers part and I'm sure some lawyer would be able to put up a case for fraud.. thats not what we're talking about tho. If it was required legitimately that would be different but in which case you could make appropriate direct or indirect deductions to your costs. There is no limit to how long a DoS attack can last. And your provider has no incentive to trace/filter if he gets a major profit if he can just make that attack last a few more hours. Indeed, and I'd be annoyed if my provider deliberately allowed this to happen, I'd probably shut down my connection to them and find some relevant contractual clause before demanding credit or legal action. I cant imagine they'd last too long doing this to everyone! That said however, my own experience of big providers (no names but one of whose name has been praised quite a lot recently on this list) is that their abuse team were completely useless. By definition, anything two parties agree to with full knowledge is fair to both of them. How DoS attacks are handled should be part of the negotiation of any ISP/customer agreement. However, for many of the contracts I've seen the contract was silent and ambiguous. True, but this is the nightmare legal world we're in, DoS attacks have tended not to disrupt billing and we assume we wont be charged but you're right, these days you have to explicitly mitigate for all possibilities.. For a 95 percentile agreement, it's reasonable for the customer to take responsibility for DoS traffic until he makes a request to the provider's NOC. It's also reasonable for the provider to charge a fixed 'incident fee' for each attack that requires NOC and network resources. It is not reasonable for the incentive structure to reward the NOC for doing nothing and penalize them for any attempt to help. Sounds like the start for a whole new discussion topic.. :) Steve
Re: Measured Internet good v. bad traffic
I can have some sympathy for the customer in this case...But... Do you consider the definition of 'bad traffic to include spam? To me, this is really simple. (as usual, IANAL, BUT...) It is 'theft of services' on the part of: a) the person(s) who wrote and released the virus, and b) contributory negligence on the part of anyone who didn't patch their systems when they found out. It would remain an open legal question if the ISP could be held negligent for not blocking the ports. Not ground I, as an ISP, would like to see explored either. Even though we did block all the appropriate ports. As to billing credit, it is an interesting problem. An equivalent would be someone causes your power utilization to go up. You still have to pay the bill. If you can prove who is doing it, you might be able to re-coup some of the costs. This all comes, again, back to the matter of enforcment for the crimes. And LEO's being unwilling to do anything unless you can show a direct financial loss. Well, the financial loss is starting to show up. Complain to your upstream, and call the long arm of the law. Bob Raymond, Steven wrote: Have received complaints from usage-based-billing Internet customers lately about not wanting to pay for the nuisance traffic caused by worm-of-the-day. I believe that in the case of a short-duration, targeted attack that can be eventually be stopped, a billing credit is probably appropriate. But what about these current plagues that go on for weeks or forever- what is your network's response? Some simply want the traffic filtered in our routers- permanently. That is my least favorite option. Others want to simply not be billed for bad traffic. My reaction is to suggest that metered billing is probably not for you, then. But I could of course sympathize if I were footing the bill. What are other network operators doing about this issue, if it is an issue for them at all? Thanks
Measured Internet good v. bad traffic
Have received complaints from usage-based-billing Internet customers lately about not wanting to pay for the nuisance traffic caused by worm-of-the-day. I believe that in the case of a short-duration, targeted attack that can be eventually be stopped, a billing credit is probably appropriate. But what about these current plagues that go on for weeks or forever- what is your network's response? Some simply want the traffic filtered in our routers- permanently. That is my least favorite option. Others want to simply not be billed for bad traffic. My reaction is to suggest that metered billing is probably not for you, then. But I could of course sympathize if I were footing the bill. What are other network operators doing about this issue, if it is an issue for them at all? Thanks
Re: Measured Internet good v. bad traffic
On Wed, 27 Aug 2003, Raymond, Steven wrote: Have received complaints from usage-based-billing Internet customers lately about not wanting to pay for the nuisance traffic caused by worm-of-the-day. I believe that in the case of a short-duration, targeted attack that can be eventually be stopped, a billing credit is probably appropriate. But what about these current plagues that go on for weeks or forever- what is your network's response? Some simply want the traffic filtered in our routers- permanently. That is my least favorite option. Others want to simply not be billed for bad traffic. My reaction is to suggest that metered billing is probably not for you, then. But I could of course sympathize if I were footing the bill. What are other network operators doing about this issue, if it is an issue for them at all? Well imho the simple way to look at this is that short bursts are generally swallowed up by the network and the upstreams and not charging is fine however for sustained traffic .. days or weeks or forever its different, if you didnt charge any customer for the increased bandwidth and load then you have to foot the cost of the network and equipment upgrades and that is surely wrong? I mean if the traffic were unrealistically to increase so that bad traffic was 50% of all traffic we would all have to double our circuit and router capacity and you either pass that cost on directly (charge for extra usage) or indirectly (increase the $ per Mb) to the user. I think you're right to say that if thats not acceptable to the user then usage based billing should be avoided for them but ultimately they will still incur the cost as you increase prices over time to foot the cost of increasing overheads. Steve
RE: Measured Internet good v. bad traffic
I mean if the traffic were unrealistically to increase so that bad traffic was 50% of all traffic we would all have to double our circuit and router capacity and you either pass that cost on directly (charge for extra usage) or indirectly (increase the $ per Mb) to the user. I think you're right to say that if thats not acceptable to the user then usage based billing should be avoided for them but ultimately they will still incur the cost as you increase prices over time to foot the cost of increasing overheads. Analogically, imagine if Burger King kept getting shipments of buns that they didn't want but still had to pay for. Their customers would get pretty pissed if BK added an 'unwanted bun' charge to their bill (absent specific prior agreement). I pay for the food I order, not the food BK's suppliers ship to BK. Of course, it's reasonable for BK to raise their prices for the costs of having to deal with the unwanted food. I sympathize with the customer. There is no reason he should pay for traffic he did not request and does not want. If unwanted traffic raises your cost of providing the service for which you are paid (providing wanted traffic) then you should raise your rates. In principle, one could certainly enter into an agreement where the customer agrees to bear the costs of unwanted traffic in exchange for a lower rate. But I certainly wouldn't assume the customer agreed to pay for traffic he doesn't want and didn't ask for unless the contract explicitly says so. And for those people entering into contracts, make sure the contract is clear about what happens with DoS attacks and where the billable traffic is measured. Otherwise you might be pretty surprised if you get a bill for 250Mbps of traffic when you contracted for a 45Mbps circuit. For those dealing with contracts already in place, if your provider argues that you are responsible for all attack traffic no matter what, ask them if that means you could possibly get billed for 1Gbps of traffic even though you only bought a T1. DS
