Re: estimation of number of DFZ IPv4 routes at peak in the future
On Sat, Mar 12, 2011 at 7:27 PM, William Herrin b...@herrin.us wrote: That must be my mistake then, because I thought the exercise was building it in a way that it stays built for the maximum practical number of years. When it has to be touched again (or tweaked if it So when you upgrade a device, you always buy the suitable device which has the highest capabilities? You put in a top-of-rack switch with 10GbE for servers with no 10GbE ports and no plans of needing 10GbE connectivity to the next round of servers? You buy a modular router for branch offices that have only a few workstations and no predictable need for upgraded connectivity? This is a good way to waste money, and a bad way to ensure that you will have the *features* you may want in the future. New features will not be back-ported to your box of choice, but you will have sunk unnecessary budget resources into that box, making it harder to justify upgrades. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Internet Edge Router replacement - IPv6 route tablesizeconsiderations
On Thu, Mar 10, 2011 at 10:51 PM, George Bonser gbon...@seven.com wrote: And I say making them /127s may not really make any difference. Say you make all of those /127s, at some point you *are* going to have a network someplace that is a /64 that has hosts on it and that one is just as subject to such an attack. If you are a content provider, it doesn't make any difference if they take down the links between your routers or if they take down the link that your content farm is on. The end result is the same. You have managed to re-arrange the deck chairs. Have another squeeze at that water balloon. Again, this is the argument put forth by the RFC wavers, that you can't solve the problem because you must want to configure /64s for ... what, exactly? Oh, right, SLAAC. More on that below. If I'm a content provider, I don't have to configure a /64 for my content farm. I can configure a /120 or whatever subnet size is practical for my environment. I can also use link-local addressing on my content farm LANs and route subnets to my content boxes, if that is somehow more practical than using a smaller subnet. If you are a service provider where practically all of your links are point to points, sure. No, you can avoid configuring /64s if you don't need SLAAC. Who needs SLAAC? I don't. It has absolutely no place in any of my environments. It seems to me that DHCPv6 will do everything which SLAAC does, and everything SLAAC forgot about. The complexity argument is pretty much indefensible when the trade-off is configuring DHCPv6 vs turning a bunch of router knobs and hoping no one ever targets your LANs with an NDP DoS. We didn't need much more host addressing, we needed more subnet addressing. I would have settled for 16 bits of host addressing and 112 bits of subnet addressing and I suppose nothing prevents me from doing that except none of the standard IPv6 automatic stuff would work anymore. None of that standard IPv6 automatic stuff works today, anyway. The state of IPv6 support on end-user CPE generally ranges from non-existent to untested to verified-to-be-broken. This is the only place in your network where /64 can offer any value, and currently, CPE is just not there. Even the latest Cisco/Linksys CPE does not support IPv6. Sure, that'll change; but what won't change is the total lack of any basis for configuring /64 LANs for content farms or any similar non-end-user, non-dynamic segments. I don't want 16 bits of host addressing. I want to choose an appropriate size for each subnet. Why? Because exactly zero of my access routers can handle 2**16 NDP entries, let alone 2**16 entries on multiple interfaces/VLANs. I would like to see much larger ARP/NDP tables in layer-3 switches, and I think vendors will deliver on that, but I doubt we'll soon see even a 10x increase from typical table sizes of today. VPS farms are already pushing the envelope with IPv4, where customers are already used to conserving addresses. Guess what, customers may still have to conserve addresses with IPv6, not because the numbers themselves are precious, but because the number of ARP/NDP entries in the top-of-rack or distribution switch is finite. And again, are you talking about all the way down to the host subnet level? I suppose I could configure server farms in /112 or even /96 (/96 has some appeal for other reasons mostly having to do with multicast) but then I would wonder how many bugs that would flush out of OS v6 stacks. I'm not getting reports of problems with smaller-than-/64 subnets from customers yet. Am I confident that I never will? No, absolutely not! Like almost everyone else, I have some customers who have configured IPv6, but the amount of production traffic on it remains trivial. That is why I allocate a /64 but provision a /120 (or similar practical size.) I can grow the subnet if I have to. I do know that /64 LANs will cause me DoS problems, so I choose to work around that problem now. If /120 LANs cause me OS headaches in the future, I have the option to revise my choice with minimal effort (no services get renumbered, only the subnet must grow.) Why would you suggest /96 as being more practical than /64 from the perspective of NDP DoS? Again, this is an example of the in-between folks in these arguments, who seem not to fully understand the problem. Your routers do not have room for 2**(128-96) NDP entries. Typical access switches today have room for a few thousand to perhaps 16k, and typical bigger switches are specifying figures below 100k. This doesn't approach the 4.3M addresses in a /96. In short, suggesting /96 is flat out stupid -- you get the worst of both worlds, potential for OS compatibility issues, AND guaranteed NDP DoS vulnerability. passing traffic. That doesn't protect against rogue hosts but there might be ways around that, too, or at least limiting the damage a rogue host can cause. How do you suggest we limit the damage a
Re: Internet Edge Router replacement - IPv6 route tablesizeconsiderations
On Fri, Mar 11, 2011 at 1:07 PM, valdis.kletni...@vt.edu wrote: Feel free to explain how SLAAC should work on a /96 with 32 bits of host address (or any amount smaller than the 48 bits most MAC addresses provide). Remember in your answer to deal with collisions. Why should SLAAC dictate the size of *every subnet* on the IPv6 Internet? This is what people who I label IPv6 Fundamentalists wish to do. They refuse to admit that their ideas were conceived in the mid-90s, that technology has advanced a great deal since that time, and ARP/NDP is a real limit now, while VLSM is no longer a tough challenge (vendors have had a couple decades to make it work really well!) I think there are a lot of people who throw around the SLAAC argument like it's actually good for something. Do these people know what SLAAC does? For core networks, it doesn't do anything. For hosting/datacenter networks and cluster/VPS environments, again, it doesn't do anything. Zero benefit. You probably don't configure these things using DHCP today. Wait, you do? Oh, it's a good thing we've got DHCPv6, which clearly can run alongside your DHCP for IPv4. Is SLAAC for end-user access networks? Not so much. See recent discussions on this list about things which are not included in SLAAC that DHCPv6 does do today. SLAAC can provide an advantage if you can live without those things, but that advantage is limited to one thing: the subnet doesn't need a DHCPv6 server (or proxy/forwarding of packets to same.) IPv4 has gotten along just fine for a long time with both full-featured and light-weight DHCP servers, and statically configured subnets. Is SLAAC solving any problem? Sure, for some situations, like SOHO networks, it's a nice option, but it's just that, an option. It isn't needed. Is SLAAC for fully peer-to-peer networks, with no central gateway? No. To function, SLAAC requires an RA message from something that decides it is a router. It isn't going to facilitate a headless, ad-hoc network to support the next revolution with peer-to-peer cell phones. So what we know is that the sole arguments from IPv6 Fundamentalists in favor of /64 LANs are * VLSM is hard (it isn't; vendors are really good at it now, otherwise IPv4 wouldn't work) * SLAAC needs it to work (not all LANs need SLAAC) * it's the standard (more on this below) I believe everything except the it's the standard argument is fully and completely debunked. If anyone disagrees with me, feel free to correct me, or argue your point until you are blue in the face. I have often been reminded that I should have been more vocal about this matter 10+ years ago, but frankly, I thought vendors, large ISPs, veterans with more public credibility than myself, or the standards folks themselves, would have straightened this out a long time ago. If you can decide for yourself that VLSM is easy and you trust your vendor to do it right (if you don't, summarize to /64 towards your core, or do one great thing IPv6 allows us to do, and summarize to *even shorter* prefixes towards your core, and carry fewer routes in core) then you are half-way there. If you realize SLAAC isn't a tool for your VPS farm or on your backbone link-nets, you're all the way there. At this point, you can deploy your IPv6 without it being broken by design. The only thing broken here is the Fundamentalists, who are stuck in a mid-1990s mindset. These guys need to get out of the way, because they are impeding deployment (for those smart enough to recognize this problem) and they are creating an almost certain need for a future re-design (for those who aren't smart.) This future doesn't depend on anything except v6 actually getting deployed enough to where DDoS happens over it at any appreciable scale. In the current state of the Internet, it is certain that this problem will happen. No visible progress has been made on solving it, except by guys like myself who are happy to cry the sky is falling, configure our networks in a non-standard way, and tell the standards folks they are wrong. The Cisco knob is progress only in that Cisco recognizes customers are concerned about this problem and allow them to steer their failure mode. If the DDoS happens before vendors provide a real solution, or before standards are revised or thrown out, you can thank those of us on the sky is falling side of this argument for testing the work-around (by never having exposed ourselves to the problem in the first place.) It's one thing to say it should be redesigned. It's another matter entirely to actually come up with a scheme that doesn't suck even harder than screw it, it's a /64. This is true. I think the price of energy is continuing to rise and our future is very uncertain as a result. I don't know how to fix it. Does that mean I should keep my opinion to myself? Of course not. Recognizing a problem is the first step on the path to a solution. I imagine the same arguments taking place before VLSM
Re: Internet Edge Router replacement - IPv6 route tablesizeconsiderations
On Fri, Mar 11, 2011 at 6:33 PM, Owen DeLong o...@delong.com wrote: Yes, you can bring as much of the pain from IPv4 forward into IPv6 as you like. You can also commit many other acts of masochism. This is the problem with Fundamentalists, such as yourself, Owen. You think that fixing things which work fine (like reasonable-sized VLSM LANs for content farms) is worth introducing a DDoS vulnerability for which there is no current defense, and for which the only feasible defense is either reversing your choice and renumbering the subnet from /64 to /smaller, or waiting until your vendors supply you with patched images for your routers and/or switches. You need to move beyond this myopic view that /64 provides a benefit that is worth this kind of operational sacrifice. When vendors cough up some more knobs, I'll be right there with you, configuring /64 subnets. I've already allocated them! It's pretty easy for me to renumber my /120 subnets to /64, after all -- I don't have to update any zone files for public-facing services, or modify significant configuration for software -- I just have to reconfigure my router and host interfaces from /120 to /64. You, on the other hand, may have addresses in use all over that /64, and condensing them into a smaller subnet is guaranteed to be at least as hard as my work for growing my subnet, and may be much more difficult -- every bit as difficult as renumbering from one IPv4 block to another. Given the current state of IPv6, your Fundamentalist way introduces new problems *and* brings the old ones forward. This makes no sense, but Fundamentalists rarely do. Personally, I prefer to approach IPv6 as a way to reduce some of the more painful aspects of IPv4, such as undersized subnets, having to renumber or add prefixes for growth, limited aggregation, NAT, and more. I look forward to that when it works. As I've noticed, I have prepared to take advantage of those things as soon as the NDP issue is resolved. None of that standard IPv6 automatic stuff works today, anyway. The state of IPv6 support on end-user CPE generally ranges from As someone using SLAAC in a number of environments, I'm confused by this statement. It seems to be working quite well in many places and end-user residential networks are certainly not the only places where it is useful. Your definition of working quite well in many places is different than mine. I'll come around to your point of view when it is possible to get working IPv6 connectivity from most major end-user ISPs, and all (or close enough) the CPE being sold at Fry's and Best Buy works right. We are pretty far from that right now. This is another thing the IPv6 Fundamentalists seem to ignore. CPE support is almost non-existent, ISP support is not there (some tier-1 transit networks still have no IPv6 product!), and the major IXPs still have three orders of magnitude more IPv4 traffic than IPv6. Cogent, Level3, and Hurricane Electric still can't decide that it's in their mutual interest to exchange IPv6 traffic with each-other, and their customers don't care enough to go to another service provider, because IPv6 is largely unimportant to them. None of this stuff works today. You aren't seeing DDoS scenarios on the v6 network today because the largest IPv4 DDoS attacks are larger than the total volume of inter-domain IPv6 traffic. Most of the top-of-rack switches I'm aware of have no problem doing at least 64k NDP/ARP entries. Many won't do more than that, but, most will go at least that far. Owen, this statement is either: 1) a gross misunderstanding on your part, because you can't or don't read spec sheets, let alone test gear 2) you've never seen or used a top-of-rack switch or considered buying one long enough to examine the specs 3) your racks are about 3 feet taller than everyone else's and you blow 100k on switching for every few dozen servers 4) an outright lie, although not an atypical one for the IPv6 Fundamentalist crowd I'd like you to clarify which of these is the case. Please list some switches which fit your definition of top-of-rack switch that support 64k NDP entries. Then list how many top-of-rack switches you are currently aware of. Don't bother listing the ones you know don't support 64k, because I'll gladly provide a list of plenty more of those, than the number of switches which you find to support 64k in a ToR form-factor. For those following along at home, how many ToR switches do indeed support at least 64k NDP entries? Unlike Owen, I know the answer to this question: Zero. There are no ToR switches that support = 64k NDP table entries. Of course, I don't really mean to call Owen a liar, or foolish, or anything else. I do mean to point out that his facts are wrong and his argument not based in the world of reality. He is a Fundamentalist, and is part of the problem, not the solution. I find it interesting that you _KNOW_ that /64 LANs will cause you DoS problems and yet
Re: Internet Edge Router replacement - IPv6 route table size considerations
On Wed, Mar 9, 2011 at 9:11 PM, Chris Woodfield rek...@semihuman.com wrote: I think this is the point where I get a shovel, a bullwhip and head over to the horse graveyard that is CAM optimization... The classic problem with any sort of FIB optimization is that you can't optimize every figure on the spec sheet at once, at least not without telling lies to your customers! You can have more compact structures which require more memory accesses and clock cycles to perform look-ups, or you can have bigger structures which improve look-up speed at the expense of memory footprint. Since the market is pretty much used to everything being advertised as wire speed now, in order to continue doing look-ups at wire speed with an ever-increasing number of routes in the FIB and with entries having longer bit masks, you need more silicon -- more parallel look-up capability, faster (or parallel) memory, or optimizations which may not maintain wire speed for all use cases (cache, interleaving, etc.) As the guy making purchasing decisions, I really care about one thing: correct information on the spec sheet. You may have noticed that some recent spec sheets from Cisco include little asterisks about the number of routes which will fit on the FIB are based on prefix length distribution, which means, in effect, that such optimizations are in effect and the box should perform at a guaranteed forwarding speed by sacrificing a guaranteed number of possible routes in FIB. Relating to IPv6 forwarding in particular, this produces an interesting problem when deploying the network: the IPv6 NDP table exhaustion issue. Some folks think it's a red herring; I obviously strongly disagree and point to Cisco's knob, which Cisco will gladly tell you only allows you to control the failure mode of your box (not prevent subnets/interfaces from breaking), as evidence. (I am not aware of any other vendors who have even added knobs for this.) If you configure a /64, you are much more likely to have guaranteed forwarding speed to that destination, and guaranteed number of routes in FIB. What you don't have is a guarantee that ARP/NDP will work correctly on the access router. If you choose to configure a /120, you may lose one or both of the first guarantees. The currently-available compromise is to configure a /120 on the access device and summarize to a /64 (or shorter) towards your aggregation/core. I see nothing wrong with this, since I allocate a /64 even if I only configure a /120 within it, and this is one of the driving reasons behind that decision (the other being a possible future solution to NDP table exhaustion, if one becomes practical.) The number of people thinking about the big picture of IPv6 forwarding is shockingly small, and the lack of public discussion about these issues continues to concern me. I fear we are headed down a road where the first large IPv6 DDoS attacks will be a major wake-up call for operators and vendors. I don't intend to be one of the guys hurriedly redesigning my access layer as a result, but I'm pretty sure that many networks will be in exactly that situation. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Internet Edge Router replacement - IPv6 route table sizeconsiderations
On Thu, Mar 10, 2011 at 1:52 PM, George Bonser gbon...@seven.com wrote: What I have done on point to points and small subnets between routers is to simply make static neighbor entries. That eliminates any neighbor table exhaustion causing the desired neighbors to become unreachable. I also do the same with neighbors at public peering points. Yes, that comes at the cost of having to reconfigure the entry if a MAC address changes, but that doesn't happen often. I wouldn't bet on the router evicting a maliciously-learned dynamic NDP entry to install a static NDP entry when an interface flaps up, and if it doesn't, I wouldn't bet on that static NDP entry ever being installed until the interface flaps again. Remember, there are several possible attack methods here, one of which is a compromised or badly broken box on a connected LAN. As Richard points out, there is *no* reason to configure /64s on point-to-point links, and there are obvious disadvantages. The RFC wavers are downright stupid to suggest otherwise. As for IXP LANs, I predict that one of two things will happen: either one or more major IXPs will be subject to NDP DoS and will decide to shrink their subnet size, allowing others to follow suit; or vendors will make NDP inspection work and be configurable enough to prevent most problems. Again, Cisco has already added a knob to some platforms which allows you to steer the failure mode. Interfaces will fail regardless of what you do; the Cisco knob just lets you decide to break NDP on only the interface(s) subject to attack instead of on the entire box. In any case, I don't judge static NDP entries on IXP LANs to be a practical long-term solution. There are obvious disadvantages to that. If your network is entirely made up of backbone routers with fairly static neighbors, your strategy can certainly work with a bit of extra effort and a vendor box that doesn't do entirely crazy things. If you have customers (those pesky customers!) they may not be so comfortable having to open a ticket and feel like they are troubleshooting a problem you've caused them because you have configured a static NDP entry facing them. If you have hosting customers with servers attached to VLANs, especially in a VPS environment where IP/MAC associations may routinely change ... good luck with those static NDP entries. Obviously, some folks will continue to cite standards for something which was developed in 1997 and still isn't really working, or claim their own fix works, until they get actual IPv6 customers. Those folks are probably choosing to redesign their access layer in the future, *AFTER* they already have customers. I have been talking to smart people about this problem for nearly ten years, and I have never heard a practical solution that doesn't involve some kind of persistent sticky NDP which refuses to make discovery requests to the LAN for addresses which have never been seen from the LAN. I've also never seen a practical idea for preventing malicious hosts on the LAN from filling the table that doesn't involve NDP inspection at the access port, some kind of database (e.g. RADIUS/etc) or additional configuration in the router, or proposals that would simply change the failure mode (e.g. rate-limit knobs.) You'll notice that there have been several discussions about this on NANOG and other mailing lists, most of which include some RFC wavers, some people saying the sky is falling, and some guys in-between who think their vendor's box will fail gracefully or that NDP learning not functioning for some or all interfaces is not bad as long as the box doesn't evict busy entries. I suggest all the folks in the middle ask themselves why Cisco added a knob *just to control the failure mode.* This is a real problem, and the current, practical fix is simply to not configure /64. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
How many IPv6 BGP routes are you planning for in DFZ?
On Wed, Mar 9, 2011 at 2:19 AM, George Bonser gbon...@seven.com wrote: The ipv4-ipv6-2 CAM profile in 5.1 gives 768K v4 routes and 64k v6 routes which should be good for quite a while. That is provided you How many IPv6 BGP routes are folks typically planning for in the DFZ before a hardware upgrade is required? Here are some relevant figures (note that my script makes some minor errors but this is good enough for discussion purposes): IPv6: Unique Origin ASes seen: 3287 Examined 4705 active routes IPv4: Unique Origin ASes seen: 36707 Examined 352688 active routes Making some assumptions, let's say every active ASN in DFZ will announce a mean of 1.4 IPv6 routes (the number seen today.) If IPv6 grows from under 10% of ASNs today to 100% of ASNs in a year or two, we will see about 53k IPv6 routes in DFZ. Keep in mind that many, if not most, ASNs originating IPv6 routes today have substantially no production services on IPv6, and they may deaggregate more in the future, etc. Some folks seem to believe that not every ASN will announce routes to the DFZ. I don't think that is a safe assumption upon which to base purchasing decisions for routers which should have a life-cycle of several years. Whether or not networks *should* announce more than one route, or any routes at all, seems debatable; but when making router purchasing decisions, I don't want to tell my clients two years from now that they have to spend capital dollars on routers just to gain IPv6 FIB. I also don't want to tell them they have to filter some routes, and make any BGP customers unhappy, and live with other downfalls of that unfortunate compromise. I am certainly not deploying any boxes that will only do 64k IPv6 FIB in default-free part of my clients' networks. It certainly will work now, and will almost certainly be safe in one year. In two years, it seems a little questionable. Beyond that time-frame, it is much easier to justify new routers, as ports become cheaper and faster; but I still do not want my clients to be forced to buy new routers simply because of overly-optimistic assumptions about IPv6 DFZ size. Really, I would like vendors to make IPv4 and IPv6 FIB come from the same pool (with obviously different allocation sizes) or allow me to configure the partitioning as I see fit. This has been the case for quite a few platforms for many years. I am not comfortable guessing at whether I will first need to exceed 500K IPv4 routes (maybe we won't even see that number) or 64K IPv6 routes (this seems a virtual certainty, but when it happens is hard to say.) Once you add L3VPN into your list of concerns, your future FIB needs become even more difficult to predict. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: IPv6? Why, you are the first one to ask for it!
I guess I'll plug this Wikipedia page again: http://en.wikipedia.org/wiki/Comparison_of_IPv6_support_by_major_transit_providers -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: What vexes VoIP users?
On Mon, Feb 28, 2011 at 6:28 PM, Leigh Porter leigh.por...@ukbroadband.com wrote: Exactly the point I made earlier. POTS is simple, it does what it does and it is pretty good at it. Now, in the background, you have a whole lot of engineering. But I would trust a DMS100 far more than any of the stuff that routes IP. POTS is cheap, easy, scalable and resistant to many disasters that would soon wipe any VoIP network out. I wouldn't call DMS100 a cheap platform. The switch gear is expensive, features are expensive, floor space is expensive, training is expensive, and provisioning, for the most part, is stuck in the dark ages. Sure, it works, but to make the generalization that it's cheaper than modern VoIP switching is just incorrect. Besides that, if you have done much DMS100 ops, you are well aware that there are many (infrequent) tasks that require multi-hour outages of major DMS100 components, e.g. one of the two CMs (control plane, for unfamiliar readers.) In addition, the official maintenance procedures often don't tell you how to perform these tasks without taking the whole switch out of service. A growing number of end-users are perfectly happy with no land-line and no VoIP, relying only on cellular phone service. I'm sure that cellular is generally orders of magnitude less reliable than POTS. I'm sure most VoIP offerings are somewhere in-between. End-users are going to choose the product they want, and for many, the choice will be to save hundreds of dollars per year while sacrificing a little bit of reliability which they are unlikely to notice or miss. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Mac OS X 10.7, still no DHCPv6
On Sun, Feb 27, 2011 at 5:16 PM, Ray Soucy r...@maine.edu wrote: This seems to have upset at least one Apple engineer who dropped the NDA bomb on me; while he didn't confirm it was there, he did imply it, and it did make me have people give a second look. (I tried to get him to admit it but he's obviously been through Apple secret keeping training). If work on DHCPv6 or other common tools are obscured by NDA, and thus information is not available to potential customers, and IT departments who must plan to support those customers, Apple is at fault, not Ray or anyone else. There is a lesson for Apple here. Secrets are cool and there is often a legitimate need to keep new features under wraps until you are actually ready to ship them (competition, delays, whatever.) Somehow, I don't think Steve Jobs is going to give a presentation on DHCPv6, and I doubt Apple's decision to ship it with their OS is going to cause Microsoft or other competitors to .. do anything differently. Obscuring some things behind NDA is good for business. IPv6 matters (specific to DHCPv6 or otherwise) are not among those things, and Apple ought to take notice of this very discussion and make their intentions and progress more public, so IT departments know what to expect. Secrecy is good for business, except when it's not. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Howto for BGP black holing/null routing
On Tue, Feb 22, 2011 at 4:55 PM, Jack Carrozzo j...@crepinc.com wrote: Maybe I read your question wrong, but null-routing things at your border is often not very useful if the traffic is flooding your transit links. Most transits publish their community lists - you just need to tag the prefix you want to blackhole with the right community. This is certainly true. Although most big transit networks offer this feature today, there are some important differences in what some of them will and won't accept. Some will only learn /32s, some say they'll accept /30-/32 but nothing shorter, some will honor anything you send them. This may be undocumented. Some networks seem to have forgotten about this feature when implementing IPv6, even though it is offered for IPv4. I don't see any value in not accepting a RTBH /24 but accepting a /30. I also don't know of any platform issues which would make deploying RTBH for IPv6 BGP customers any more difficult than doing so for IPv4. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6 naysayer...)
On Fri, Feb 18, 2011 at 10:34 AM, Zed Usser zzu...@yahoo.com wrote: Reduce, yes. Remove, no. Without a global cutoff date for the IPv6 transition, it's not like IPv4 is going to disappear overnight. Furthermore, without any IPv4/IPv6 translation, the first IPv6 only networks are going to be awfully lonely. I suspect Google, Microsoft, and others have already figured out a beneficial (to everyone) way to monetize this. If I'm an ISP with working IPv6, and my competitor in a given region is an ISP without IPv6, I'd like to advertise to all the end-users of that ISP whenever they go to a search engine that sells ads. Since these search engine companies have figured out white-listing users into good IPv6, it's no great leap to suggest that they'll eventually black-list IPv4 users into bad, and tie that into their advertising system for ISPs to purchase nicely-targeted banners/links. If my ISP is reading this, please tell both your residential and business technical and sales departments to come up with a better answer than we are not going to support IPv6 because that's only for ISPs that run out of IPv4. Otherwise, I'd bet Google will be more than willing to let your competitors give customers a different answer in the near future! -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6naysayer...)
On Fri, Feb 18, 2011 at 1:14 PM, George Bonser gbon...@seven.com wrote: One thing they can do, and I would live to see some popular destination site do this, is to say something like: we have this really cool new thing we are rolling out but, sorry, it is available only via IPv6 or we will continue supporting all of today's features on v4 but all new features will be rolled out on v6 only. I doubt any top web sites or popular services will do this, because there is no commercial advantage to it. It is great to see Google, Yahoo, and other companies taking a big step by deciding to serve up by default for one day. It also should indicate to everyone how far we are from the goal line, not because Google or Yahoo aren't doing their homework, but because their end-users' ISPs aren't. If these companies are only willing to do by default for one day on a trial basis, and that with months of notice and perhaps preparation, they clearly should not be willing to make some cool new revenue-generating feature exclusive to IPv6 end-users. On Fri, Feb 18, 2011 at 1:53 PM, Franck Martin fra...@genius.com wrote: If I recall Comcast and Time Warner are participating in IPv6 day. This should create enough eyeballs to show on web analytics graph and provide the shift that makes nat444 irrelevant. I am afraid you may be a little disappointed. The number of users with IPv6-capable CPE, to say nothing of home LANs, may still be quite limited by that time. It's still progress, but I don't think anything except IPv4 depletion will increase IPv6 adoption. For a network operator I'm looking at the ipv6 ipv4 ASN ratio. Once it passes 10% we will have a snow ball effect in the core. Unfortunately, many ASNs who originate IPv6 address space have few or no functioning services on IPv6. A simple ASN ratio is not a very useful metric. You also do not have visibility into how much infrastructure is based on tunnels, whether or not v6 even reaches customer access ports or even is enabled backbone-wide, etc. I agree it is encouraging to see new ASNs originating IPv6 routes every day, but again, this is more an indicator of we got a /32 from the RIR and configured it on our router than we are using v6 in a production capacity (or are prepared to flip the switch.) I really do think that advertising may be the thing that eventually forces some end-user ISPs to get in gear. If end-users went to www.google.com on IPv6 day (and perhaps after) and got a message saying your ISP is great or here are some ISPs you might want to consider, because yours can no longer reach some Internet destinations, I suspect that would give ISPs a very serious reason to spend the necessary resources to get v6 done. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: [arin-announce] ARIN Resource Certification Update
On Sun, Jan 30, 2011 at 12:40 PM, Owen DeLong o...@delong.com wrote: Because they publish data you have signed. They don't have the ability to modify the data and then sign that modification as if they were you if they aren't holding the private key. If they are holding the private key, then, you have, in essence, given them power of attorney to administer your network. If you're OK with that, more power to you. It's not the trust model I would prefer. I suspect that many users would prefer to trust ARIN with their private keys, if offered that choice. The reasons are simple; adoption will be more wide-spread if RPKI is easier to do; and as we all know, there are an awful lot of BGP networks which are: * on auto-pilot, with no clued in-house staff and no stable relationships with outside clue * driven by people who are somewhere between totally clueless and capable of understanding public-key encryption * driven by over-worked people who simply don't have time for another to-do of any complexity Many users would benefit from the kind of hosted service that is made available by, for example, RIPE. In fact, if they felt they could trust ARIN (or any alternative service which may exist), most of my clients would be perfectly fine with such a service, and I would not advise them to do otherwise without a valid business reason and a belief that equal or superior security would be provided by not using such a hosted service. Since ARIN holds ultimate authority over the ISP's address space anyway, if ARIN's private keys become compromised, whether or not you held onto your own keys will not matter to the rest of the world. If I understand correctly, John has expressed that ARIN's concern is they may be sued if their hosted service fails to perform, and that ordinary contractual language may be unable to limit damages if the reality is that the service-customer has no other choice but to use the ARIN service. This is clearly not a legitimate concern if there is an alternative to such an ARIN hosted service, such as using no hosted service at all, or the possibility of using another one. I don't see how the lack of ARIN providing a hosted service immediately in any way prevents them from doing so in the future. If widespread RPKI adoption doesn't happen and a few more accidental or intentional YouTube black-holes do happen, it seems likely that stakeholders will encourage ARIN to do more, and a hosted service would be an obvious step to increase adoption. As you know, my comfort level with ARIN handling tasks of an operational nature is not high; but if they are going to participate in RPKI in any way, I think they should be capable of performing similarly to RIPE. If not, we should be asking ourselves either 1) why would anyone trust RIPE with their keys; or 2) why is RIPE more trustworthy than ARIN? If the answer to that is RIPE is significantly more competent than ARIN (most folks I know are of this belief) then this discussion should not be about one technical effort. Instead, it should be about how to make ARIN better. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: ARIN IRR Authentication (was: Re: AltDB?)
On Thu, Jan 27, 2011 at 10:00 PM, John Curran jcur...@arin.net wrote: Based on the ARIN's IRR authentication thread a couple of weeks ago, there were suggestions placed into ARIN's ACSP process for changes to ARIN's IRR system. ARIN has looked at the integration issues involved and has scheduled an upgrade to the IRR system that will accept PGP and CRYPT-PW authentication as well as implementing notification support for both the mnt-nfy and notify fields by the end of August 2011. I'm glad to see that a decision was made to improve the ARIN IRR, rather than stick to status-quo or abandon it. However, this response is essentially what most folks I spoke with off-list imagined: You have an immediate operational security problem which could cause service impact to ARIN members and others relying on the ARIN IRR database, and fixing it by allowing passwords or PGP to be used is not very hard. As I have stated on this list, I believe ARIN is not organizationally capable of handling operational issues. This should make everyone very worried about any ARIN involvement in RPKI, or anything else that could possibly have short-term operational impact on networks. Your plan to fix the very simple IRR problem within eight months is a very clear demonstration that I am correct. How did you arrive at the eight month time-frame to complete this project? Can you provide more detail on what CRYPT-PW hash algorithm(s) will be supported? Specifically, the traditional DES crypt(3) is functionally obsolete, and its entire key-space can be brute-forced within a few days on one modern desktop PC. Will you follow the practice established by several other IRR databases (including MERIT RADB) and avoid exposing the hashes by way of whois output and IRR database dumps? If PGP is causing your delay, why don't you address the urgent problem of supporting no authentication mechanism at all first, and allow CRYPT-PW (perhaps with a useful hash algorithm) and then spend the remaining 7.9 months on PGP? The plan and schedule you have announced is indefensible for an operational security issue. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: IPv6 prefix lengths
Richard's employer is exactly the kind of organization that has not been able to effectively multi-home their discrete branch-offices on the IPv4 Internet, because RIR allocation policy set the bar for receiving IPv4 addresses for those small locations just high enough to steer us away from that feature or problem, depending on how you look at it. If every Richard Barnes announces a few dozen /48s into the global BGP table, it will not be long before the 300k+ IPv4 BGP table looks neat and organized, and the CIDR Report will come out each week with a message begging e.g. Starbucks to aggregate their coffee shop wireless hot-spots, instead of shaming Bell South for having a large number of de-aggregates for their substantial ISP business. Most people do not know about the multi-homing feature designed into IPv6. Most people who do, seem to agree that it may not see enough practical use to have meaningful impact on routing table growth, which will no longer be kept in check by a limited pool of IP addresses and policies that make it a little difficult for a very small network to become multi-homed. This may be another looming IPv6 headache without a sufficient solution to set good practices now, before deployment sky-rockets. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB? (IRR support direction at ARIN)
On Mon, Jan 10, 2011 at 12:37 PM, Jon Lewis jle...@lewis.org wrote: On Sun, 9 Jan 2011, Charles N Wyble wrote: I am simply suggesting it is dangerous and irresponsible to run an IRR with only MAIL-FROM authentication, and quite easy to also support CRYPT-PW. ARIN should either support passwords or immediately make The trouble is, since the DES crypt passwords are publicly accessible, even CRYPT-PW is not much security. I suspect with a copy of the db, a passsword cracking program, and some modest computing capacity, you could crack all DES crypt() is not completely trivial yet, but I agree, it is far from state-of-the-art. It is substantially superior to MAIL-FROM. In addition, MERIT reduced this problem by simply filtering out the hashes from the RADB.db file and whois output (and presumably also, the www.radb.net tools.) -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB?
On Sun, Jan 9, 2011 at 1:09 PM, John Curran jcur...@arin.net wrote: Please suggest your preferred means of IRR authentication to the ARIN suggestion process: https://www.arin.net/participate/acsp/index.html Alternatively, point to a best practice document from the operator community for what should be done here. ARIN's work plan is very much driven by community input, so that's what is needed here. John, I appreciate you taking time to respond to this while on vacation. However, I think we all know that your response is not a here is how you tell us what to do, it's a here is our cop-out response to make an incredibly simple fix either never happen, or take six months to make it through the ARIN process. If you truly do not understand the posts regarding this matter, I will summarize them for you very simply: 1) ARIN IRR is a tool that has operational impact; service providers use it to build prefix-lists automatically, and if the data that underlies those prefix-lists is corrupted, networks that use the ARIN IRR will see their transit providers stop accepting their BGP announcements overnight. This is not a some database might be inaccurate but it's okay, problem; it is an operational problem. Some peoples' networks depend on that data not becoming corrupted. Specifically, every network that uses ARIN IRR. 2) ARIN IRR has effectively no security for record updates or deletes. Anyone who knows how to forge an email From: header can corrupt or delete part or all of the ARIN IRR database at any time. ARIN IRR is the only database that I am aware of without support for at least password authentication. The standard toolset supports passwords trivially. 3) If not supporting passwords was a business-driven decision, it was a bad one, but perhaps a mistake born out of ignorance. If it was a technically-driven decision by the staff members responsible for implementing and maintaining the ARIN IRR, those staff members are not qualified to handle anything of an operational nature, and you would be well-advised to find jobs for them that don't require any attentiveness to operational security. 4) The ARIN process will almost certainly not be the route taken when a change eventually arises. Some black hat will eventually decide it would be a clever prank to erase or corrupt the entire database, and you will then be faced with three choices; a) implement passwords immediately and not allow any updates from users who haven't selected one; b) make the ARIN IRR read-only and effectively make it useless; c) ignore the problem, at which point no ISPs will be willing to mirror the ARIN IRR anymore, because its data is a liability, not an asset. I appreciate that there is a process to go through for proposing ARIN policy changes, etc. Your suggestion that this be used when addressing an operational security matter is foolish and provides plenty of ammo for people who say ARIN is ineffective (or worse.) I suggest you take a moment to think about what the news coverage might be if this eventually blows up in a big enough way to interest news people. If a bunch of ISPs go down overnight due to an ARIN oversight, will some savvy reporter ask himself who at ARIN knew they were running an operationally-important service with no security mechanism at all? Will he have much trouble finding out about a mailing list discussion in which the CEO of ARIN glazed over the issue and referred a whistle-blowing person to the ARIN policy process? Will he then ask if ARIN is an effective steward of RPKI? Will his article assign blame to you personally? Will he draw some link to Chinese interception of 15% of the Internet? Who knows how mainstream press would interpret such an event, if it was big enough to attract attention. If I were you, though, I would not want my signature at the bottom of an email essentially telling someone to go post on the correct mailing list. I suggest you don't be the ARIN CEO that gets mud in his eye because he didn't understand the value of a password over mail-from. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB? (IRR support direction at ARIN)
On Sun, Jan 9, 2011 at 6:27 PM, Randy Bush ra...@psg.com wrote: Do you: 1) want IRR services, and if so, with what features? 2) believe IRR services should be provided by ARIN? the irr is slightly useful today. so, iff it is cheap and easy, arin providing an open and free instance is a public good. again, iff it is easy and cheap. and please do not waste time trying to 'fix' the irr, sad to say it's trying to make a silk purse out of a sow's ear. I'm not suggesting that ARIN undertake a large and complex effort to solve a bunch of issues with IRR. All I am suggesting is that they prevent anonymous bad guys with no inside information, special access, or knowledge of passwords, from corrupting the data which some networks choose to publish in ARIN IRR. I am simply suggesting it is dangerous and irresponsible to run an IRR with only MAIL-FROM authentication, and quite easy to also support CRYPT-PW. ARIN should either support passwords or immediately make their IRR read-only and stop offering it as a service. Imagine if there was a Slashdot article or something about this, how long would it take for some 14-year-old to erase the whole database, and how that would pretty much force ARIN to make a choice anyway, but also, create a lot of negative fall-out that might jeopardize trust in ARIN with regard to other operational matters, like RPKI. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB? (IRR support direction at ARIN)
On Sun, Jan 9, 2011 at 6:48 PM, Randy Bush ra...@psg.com wrote: jeff, i do not disagree that running an irr instance with only mail-from is s 1980s. and, as mans points out, there is free software out there to do it (i recommend irrd). but i do not see good cause for arin to spend anything non-trivial to fix a problem in an irr instance which is not used very much. i.e. better to drop it than to spend non-trivial money to modernize it. I agree that if ARIN thinks it would be too costly to support password authentication, they should make the database read-only so users will migrate away from it and no damage can be done by bad guys. but more to the point, by 'fix' it, i did not mean modernizing the auth method set. i meant the content, syntax and semantics. I understood what you meant, and again, I agree with you; there is no reason to invest a lot of time and resources in something that should be made obsolete by other work already in progress. The fix I want is simply eliminating the large liability by continuing to allow updates with MAIL-FROM authentication. I believe ARIN IRR actually does support MD5 authentication, but if you email the ARIN IRR person, or go to ARIN's web site, you are told that only MAIL-FROM is allowed. So they probably already have the appropriate technical mechanism in place AND JUST AREN'T USING IT, and are actively discouraging users from utilizing it. This would be an example of ARIN's ineffectiveness when it comes to operational matters, and is why I have real fear that RPKI may one-day be a disaster because ARIN is an ineffective steward. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB?
On Sun, Jan 9, 2011 at 7:33 PM, John Curran jcur...@arin.net wrote: My reason for responding is simply to make sure that ARIN is doing what the community wants. I won't deny that this may take some time depending on exactly what is involved, but in my mind that is far better than not fixing the situation. How will ARIN respond to operational security matters with regard to RPKI infrastructure in the future? What experience does ARIN have with operational security in the past? When faced with DNS server vulnerabilities, did ARIN solicit community feedback before patching the servers responsible for IN-ADDR.ARPA zones administered by ARIN? Or did ARIN treat this matter as a legitimate, operational security concern, and apply whatever technical solution was available and generally accepted by other organizations administering DNS servers? Why should an operational security issue with the ARIN IRR be handled as a policy issue? Do you know that I have emailed ARIN about this both recently and in years past? Am I the only person who has ever tried to bring this to ARIN's attention? I doubt that. Are the personnel managing the ARIN IRR oblivious to the fact that every other IRR database except ARIN supports at least some form of password authentication? Are these personnel qualified to handle services with operational impact? Do you, or they, know that ARIN's IRR technical infrastructure actually does support password security, and that records exist in the ARIN IRR database with MD5 authentication, but that email to ARIN about this are answered with replies that only MAIL-FROM is possible? Why does the ARIN web site make no mention of anything besides MAIL-FROM? Thanks; I'm aware of the ARIN IRR and how operators in the community make use of it, and have run ISPs which have made use of the data for route filtering. When you ran ISPs that made use of IRR data for route filtering, did you use any kind of authentication when publishing and maintaining your own records, or advise customers to use such? Did the possibility of malicious data corruption or erasure ever enter your mind? Agreed; dropping me an email is a fine process for operational security matters. Consider this one so reported. What will the process be for handling operational security issues regarding future RPKI infrastructure? It is conceivable that there may be no alternative to ARIN, in the ARIN region, for trusted routing information data in the future. Today, we can choose not to use ARIN IRR, and the huge majority of networks who publish IRR data use their ISP databases or MERIT RADB. Are we faced with the possibility that ARIN simply doesn't have personnel capable of handling operational services, yet are forcing ARIN down a road that may make them a sole source of something we all need? If so, perhaps this is a very bad idea in need of further debate. I think the mentality at ARIN is one of paper-pushers and policy guys. That's perfectly fine for an organization whose main function is ... processing paperwork and allocating IP addresses. It is perhaps a very bad idea to ask ARIN to do operational things which they are very clearly unprepared to handle, to such an extent that they may need additional or different personnel, and really need to change their mentality. I understand that the technical side of the RPKI implementation at ARIN is most likely entrusted to Paul Vixie and ISC, which is a good thing. I never read an email from Paul saying, I think we need to solicit feedback before we patch this BIND issue. DNSSEC progress has taken a very long time, but that hasn't stopped ISC from continuing to provide quick technical solutions to immediate technical problems. What really worries me is ... if there is some serious issue with RPKI infrastructure in the future, will ARIN be able to solve it in an operational time-frame, or won't they? -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB?
On Sun, Jan 9, 2011 at 10:47 PM, John Curran jcur...@arin.net wrote: Jeff - ARIN does indeed have folks who worry about whether the policy development process is being followed. We also have folks who actually implement the policy and issue number resources. And we all agree that this is ARIN's primary role, and what ARIN, organizationally, has been built to be good at. This is what members consider when electing the BoT and no doubt drives ARIN's day-to-day business and technical decisions. is that we also have quite a few folks who have run production operational services both for the Internet and other mission-critical environments. What does ARIN, as an organization, do that has short-term operational impact on its members? Two things that I am aware of: IN-ADDR.ARPA delegation and IRR. One of these things gives people no reason to complain. The other is demonstrably insecure in a manner that could have really serious, and embarrassing, consequences, both financial for the members, and in terms of peoples' confidence in ARIN. I'm not surprised that the IRR allows plaintext passwords, but am myself stunned if indeed we require them, since that disallows even a modicum of protection from trivial acts of sabotage. Rather than repeat what lack of information there is on the web site in regards to what forms of IRR authentication is available, I will go determinate the state of reality and post back here asap. At a minimum, we need much clearer documentation, but if more is required, we'll get it fixed asap. Thanks, I am glad you are now looking into this. To be clear, it's not just plain text passwords. There aren't any passwords for the majority of objects. The ARIN documentation indicates that only MAIL-FROM is supported. When asked about this, ARIN personnel who respond to rt...@arin.net reply that yes, MAIL-FROM is the only authentication mechanism supported, and that no, there is no support for passwords (good) or PGP (also good, but too complicated for some users.) This isn't simply an issue of plain text passwords. Your mechanism is MAIL-FROM, which means the only check that is done on update/add/delete requests is the From: header. The ARIN database, which is publicly mirrored, contains the email addresses that must be used to add/update/delete objects maintained by a given mntner: object. All you have to do to corrupt or erase a record is look up the record you want to corrupt in the IRR, then look up that mntner, then forge an email from the auth: MAIL-FROM listed in that mntner record. It's dead simple and it is not plain text passwords, it is no passwords at all. The reason I am still posting is I am deeply concerned about the lack of technical and management competence needed to let this happen in the first place. You shouldn't seriously believe that no ARIN staffer ever thought about this, while also believing that ARIN is currently capable of administering RPKI, by its very nature and as its primary goal, to improve operational network security. For this reason, I think your true task is not simply to address the IRR issue, but to change the mentality at ARIN. If you do have technically skilled personnel, something is preventing them from being effective. If there isn't a management or cultural problem stopping folks from speaking up, then, quite frankly, I think you may be greatly over-estimating the technical savvy of ARIN staff. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB?
On Sat, Jan 8, 2011 at 2:47 PM, Christopher Morrow morrowc.li...@gmail.com wrote: I don't think rr.arin.net and RPKI have anything to do with each other. I think the direction the RPKI should/is taking is to have the I at least think that whatever future and time-table is planned for RPKI, this should not stand in the way of ARIN offering an effective authentication mechanism for the ARIN IRR. FYI, the reply I received from ARIN was that there are no plans to improve its authentication capability. I didn't ask why and don't really care why it has never had anything more than MAIL-FROM in the past. Either it should be improved (IMO) or it shouldn't be. I really do wonder what ARIN's plan is if a bad guy decides to forge emails and delete or modify some or all of the objects. Would they just shut it down, improve authentication, or keep doing business as usual? I am always surprised that black hat folks do not do things like this when faced with a damaging vulnerability that can easily be exploited with no way to trace the activity back to the bad guy. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Thu, Jan 6, 2011 at 2:42 AM, Joel Jaeggli joe...@bogus.com wrote: icmp6 rate limiting both reciept and origination is not rocket science. The attack that's being described wasn't exactly dreamed up last week, is as observed not unique to ipv6, and can be mitigated. That does not solve the problem. Your response, like most on this thread, clearly indicates that you do not understand the underlying issue, or how traffic is actually forwarded. Neither IPv6 or IPv4 packets are simply forwarded onto the Ethernet, which is why the ARP/NDP table resource is required; a mapping from layer-3 to layer-2 address is needed. If the table resource for these entries is exhausted, no new mappings can be learned, and bad things happen. Either hosts on the specif interface, or the entire box, can no longer exchange traffic through the router. If an artificial rate-limit on discovery traffic is reached, discovery of mappings will also be impeded, meaning the denial-of-service condition exists and persists until the attack ceases. This may also affect either just that interface, or all interfaces on the router, depending on its failure mode. Rate-limiting discovery traffic still breaks the attached LANs. How badly it breaks them is implementation-specific. It does avoid using up all the router's CPU, but that doesn't help the hosts which can't exchange traffic. Again, depending on the router implementation, the fraction of hosts which cannot exchange traffic may reach 100%, and in effect, the router might as well be down. You can still have this problem when you assign a bunch of /112s how many neighbor unreachable entries per interface can your fib hold? You are correct, but the device can hold a significant number of entries compared to the size of a /112 subnet, just like it can hold a significant number of v4 ARP entries compared to a v4 /22 subnet. The difference is, ARP/NDP mappings for one /64 subnet can fill all the TCAM resources of any router that will ever exist. This is why more knobs are needed, and until we have that, the /64 approach is fundamentally broken. Again, until this problem is better-understood, it will not be solved. Right now, there are many vulnerable networks; and in some platforms running a dual-stack configuration, filling up the v6 NDP table will also impact v4 ARP. This means the problem is not confined to a cute beta-test that your customers are just starting to ask about; it will also take out your production v4 network. If you are running a dual-stack network with /64 LANs, you had better start planning. It's not just a problem on the horizon, it's a problem right now for many early-adopters. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Thu, Jan 6, 2011 at 7:34 AM, Robert E. Seastrom r...@seastrom.com wrote: I continue to believe that the allocate the /64, configure the /127 as a workaround for the router vendors' unevolved designs approach, As a point of information, I notice that Level3 has deployed without doing this, e.g. they have densely packed /126 subnets which are conceptually identical to /30s for infrastructure and point-to-point. I am still taking your conservative approach, as I see no operational disadvantage to it; but opinions must differ. On Thu, Jan 6, 2011 at 11:28 AM, valdis.kletni...@vt.edu wrote: And the ZOMG they can overflow the ARP/ND/whatever table is a total red herring - you know damned well that if a script kiddie with a 10K node botnet wants to hose down your network, you're going to be looking at a DDoS, and it really doesn't matter whether it's SYN packets, or ND traffic, or forged ICMP echo-reply mobygrams. I agree, botnets are large enough to DDoS most things. However, with the current state of ARP/ND table implementations in some major equipment vendors' routers, combined with standards-compliant configuration, it doesn't take a botnet. I could DoS your subnet (or whole router) without a botnet, say, using an IPv6 McDonald's Wi-Fi hotspot. That is very much a legitimate concern. A few hundred packets per second will be enough to cripple some platforms. On Thu, Jan 6, 2011 at 1:20 PM, Owen DeLong o...@delong.com wrote: And there are ways to mitigate ND attacks as well. No, Owen, there aren't. The necessary router knobs do not exist. The Cisco approach is currently to police NDP on a per-interface basis (either with per-int or global configuration knob) and break NDP on the interface once that policer is exceeded. This is good (thanks, Cisco) because it limits damage to one subnet; but bad because it exemplifies the severity of the issue: the Cisco solution is known to be bad, but is less bad than letting the whole box break. Cisco is not going to come up with a magic knob because there isn't any -- with the current design, you have to pick your failure modes and live with them. That's not good and it is not a Cisco failing by any means, it is a design failing brought on by the standards bodies. I would also like to reply in public to a couple of off-list questions, because the questions are common ones, the answers are not necessarily cut-and-dry (my opinion is only one approach; there are others) and this is the kind of useful discussion needed to address this matter. I will leave out the names of the people asking since they emailed me in private, but I'm sure they won't mind me pasting their questions. Anonymous Questioner: What do you think about using only link-local ip addresses on the infrastructure links please? I can't think of any potential drawbacks do you? This can be done, but then you don't have a numbered next-hop for routing protocols. That's okay if you re-write it to something else. Note that link-local subnets still have an NDP table, and if that resource is exhausted due to attacks on the router, neighbors with link-local addressing are not immune. link-local scope offers numerous advantages which are mostly out-weighed by more practical concerns, like, how hard it is going to be to convince the average Windows sysadmin to configure his machine to suit such a design, instead of just taking his business elsewhere. Not a problem for enterprise/gov/academia so much, but a problem for service providers. On Thu, Jan 6, 2011 at 3:43 PM, Jack Bates jba...@brightok.net wrote: Given that the incomplete age is to protect the L2 network from excessive broadcast/multicast, I agree that aging them out fast would be a wiser I agree that it would be nice to have such a knob. I bet you and I would make different choices about how to use it, which is the whole point of having a knob instead of a fixed value. I'm still a proponent for removing as needed requests like this, though. It would have been better to send a global everyone update me request periodically, even if triggered by an unknown entry, yet limited to only broadcasting once every 10-30 seconds. Given that all requests for an unknown arp/ND entry results in all hosts on the network checking, it only makes sense for all hosts to respond. There This isn't a new idea and it has its own set of problems, which are well-understood. IPv6 NS messages are more clever than ARP, though, and are sent to a computed multicast address. This means that the number of hosts which receive the message is minimized. See RFC2461 section 2.3 for the quick introduction. NDP is better than ARP. However, your statement that NDP has all (I'd like to say some) of the same problems as ARP but the increased subnet size has magnified them, is basically correct. NDP does some things a lot better than ARP, but not this. It's important to realize that when this stuff was designed, there were few hardware-based layer-3
Re: IPv6 - real vs theoretical problems
On Thu, Jan 6, 2011 at 5:00 PM, Deepak Jain dee...@ai.net wrote: As far as I can tell, this crippling of the address space is completely reversible, it's a reasonable step forward and the only operational loss is you can't do all the address jumping and obfuscation people like to talk about... which I'm not sure is a loss. I largely agree with you, but my knowledge is similar to yours, and does not extend to dealing with low-end CPE, dormitory LANs, hot spots, or mobile networks. I am by no means an authority in those areas and I remain open to the possibility that there may be some operational advantages to the IPv6 addressing concept for those users. The problem is there are very serious operational disadvantages for you and I, but the standard tells us to do it anyway. I would like the hot spot or mobile guys to be able to choose /64 if they want to. I need to choose otherwise, and customers expecting /64 as the standard are going to be disappointed until the standard allows for different choices. I don't have an opinion on whether the address space is truly crippled. If I did, I'm not sure it would be useful. Classful addressing ran out of gas in IPv4, so IPv6 has a huge number of bits to hopefully avoid a repeat of that. Okay, I can buy into that. There are some major networks who aren't, though, and I think they made a very conscious choice. We won't know if it's a necessary choice for a long time. I will choose to devote my arguing-on-the-mailing-list time to topics I think are more useful to discuss. I do not think you will change very many minds about IPv6 numbering schemes. I hope I will change some minds about the current safety of public /64 LANs and get more folks talking to their vendors about it, which should give us some kind of solution sooner, rather than later. Choose your battles. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: IPv6 - real vs theoretical problems
On Thu, Jan 6, 2011 at 8:04 PM, Jimmy Hess mysi...@gmail.com wrote: It is advisable to look for much stronger reasons than With IPv4 we did it or With IPv4 we ran into such and such problem due to unique characteristics of IPv4 addressing or other IPv4 conventions that had to continue to exist for compatibility reasons, etc, etc. I certainly agree that there are many advantages to the greater address space offered by IPv6. I don't mean to advocate that we do things the same way as was necessary to conserve v4 address space, and I'm sure we all realize that RIR policies necessarily contributed to routing table growth in trade for extending the life of the available address space. I'm not blindly deploying /64 networks, either. Doing so with the current set of problems, and lack of knobs, is very foolish. My transit providers offer a mix of /126 and /124 demarc subnets so far, and /124 is what I choose to standardize on for my BGP customers and private peering, for simplicity and convenience. As I mentioned before, I currently allocate a /64 and configure a /124, so I am not painting myself into a corner either way. How many of us with an appreciable level of expertise remain concerned that our approach may need significant adjustment? How many think we know what those potential adjustments may be, and have planned to make them easy (or transparent) for ourselves and customers if they become necessary? This is what is IMO most important to a responsible IPv6 deployment. To do otherwise is inviting unpredictable future pain. I am comforted by the fact that Level3 is deploying customer demarc subnets as /126 and is NOT allocating a /64 for each, but are instead packing them densely in an IPv4 /30 fashion. They recognize problems with the /64 approach, choose not to follow the standard to the letter, and implement their dual-stack network in a way they presumably believe is safe and scalable. Large networks like Level3 choosing to insist that equipment vendors support this configuration, rather than have problems with densely packed subnets smaller than /64, will mean that anyone who wants to sell a router to Level3 had better make it work correctly this way, which is good for the small guy like me who thinks he will eventually transition to that configuration. Right now, I am still hedging my bet. Are there any large transit networks doing /64 on point-to-point networks to BGP customers? Who are they? What steps have they taken to eliminate problems, if any? -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Thu, Jan 6, 2011 at 8:47 PM, Owen DeLong o...@delong.com wrote: 1. Block packets destined for your point-to-point links at your borders. There's no legitimate reason someone should be Most networks do not do this today. Whether or not that is wise is questionable, but I don't think those networks want NDP to be the reason they choose to make this change. 2. For networks that aren't intended to receive inbound requests from the outside, limit such requests to the live hosts that exist on those networks with a stateful firewall. Again, this doesn't fix the problem of misconfigured hosts on the LAN. I can and shall say it over and over, as long as folks continue to miss the potential for one compromised machine to disable the entire LAN, and in many cases, the entire router. It is bad that NDP table overflow can be triggered externally, but even if you solve that problem (which again does not require a stateful firewall, why do you keep saying this?) you still haven't made sure one host machine can't disable the LAN/router. 3. Police the ND issue rate on the router. Yes, it means that an ND attack could prevent some legitimate ND requests from getting through, but, at least it prevents ND overflow and the working hosts with existing ND entries continue to function. In most cases, this will be virtually all of the active hosts on the network. You must understand that policing will not stop the NDCache from becoming full almost instantly under an attack. Since the largest existing routers have about 100k entries at most, an attack can fill that up in *one second.* On some platforms, existing entries must be periodically refreshed by actual ARP/NDP exchange. If they are not refreshed, the entries go away, and traffic stops flowing. This is extremely bad because it can break even hosts with constant traffic flows, such as a server or infrastructure link to a neighboring router. Depending on the attack PPS and policer configuration, such hosts may remain broken for the duration of the attack. Implementations differ greatly in this regard. All of them break under an attack. Every single current implementation breaks in some fashion. It's just a question of how bad. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Thu, Jan 6, 2011 at 9:31 PM, Owen DeLong o...@delong.com wrote: You must understand that policing will not stop the NDCache from becoming full almost instantly under an attack. Since the largest existing routers have about 100k entries at most, an attack can fill that up in *one second.* If the policing rate is set to ~100 requests per second, or, even 1000 requests per second, then, I'm not sure why you think this. With a 100pps policer, it is trivial for an attack to make its NS requests far more likely to make it through the policer than legitimate NS requests that would result in discovering a valid layer-2 mapping. If you are hitting the policer, the subnet is broken. If you don't have a policer, the table is full and ... the subnet is broken. See how it's a problem that isn't solvable with a simple policer? Note that the Cisco solution is indeed a configurable per-interface policer, which is better than nothing, but does not fully solve the problem. Policing isn't a new idea. I'm not sure it's a step in the right direction, or just prolonging an inevitable change towards a real fix. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Thu, Jan 6, 2011 at 9:24 PM, Joe Greco jgr...@ns.sol.net wrote: With today's implementations of things? Perhaps. However, you show yourself equally incapable of grasping the real problem by looking at the broader picture, and recognizing that problematic issues such as finding hosts on a network are very solvable problems, and that we are at an early enough phase of IPv6 that we can even expect some experiments will be tried. Look beyond what _is_ today and see if you can figure out what it _could_ be. There's no need for what I suggest to DoS a router; that's just accepting a naive implementation and saying well this can't be done because this one way of doing it breaks things. It is better to look for a way to fix the problem. Actually, unlike most posters on this subject, I have a very good understanding of how everything works under the hood. For this reason, I also understand what is possible given the size of a /64 subnet and the knowledge that we will never have adjacency tables approaching this size. If you are someone who thinks, oh, those Cisco and Juniper developers will figure this out, they just haven't thought about it hard enough yet, I can understand why you believe that a simple fix like no ip directed-broadcast is on the horizon. Unfortunately, it is not. The only thing they can do is give more mitigation knobs to allow operators to choose our failure modes and thresholds. To really fix it, you need a smaller subnet or a radical protocol change that will introduce a different set of problems. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Wed, Jan 5, 2011 at 3:31 AM, Mohacsi Janos moha...@niif.hu wrote: Do you have some methods in your mind to resolve ARP/ND overflow problem? I think limiting mac address per port on switches both efficient on IPv4 and IPv6. Equivalent of DHCP snooping and Dynamic ARP Inspection should be implemented by the switch vendors But remember DHCP snooping et al. implemented in IPv4 after the first serious attacks...Make pressure on your switch vendors Equipment vendors, and most operators, seem to be silent on this issue, not wishing to admit it is a serious problem, which would seem to be a required step before addressing it. Without more knobs on switches or routers, I believe there are only two possible solutions for production networks today: 1) do not configure /64 LANs, and instead, configure smaller subnets which will reduce the impact of scanning attacks This is not desirable, as customers may be driven to expect a /64, or even believe it is necessary for proper functioning. I brought this up with a colleague recently, who simply pointed to the RFC and said, that's the way you have to do it. Unfortunately, configuring the network the way the standard says, and accepting the potential DoS consequences, will likely be less acceptable to customers than not offering them /64 LAN subnets. This is a foolish position and will not last long once reality sets in, unless vendors provide more knobs. 2) use link-local addressing on LANs, and static addressing to end hosts. This prevents a subset of attacks originated from the Internet, by making it impossible for NDP to be initiated by scanning activity; but again, is not what customers will expect. It may have operational disadvantages with broken user-space software, is not easy for customers to configure, and does not permit easy porting of addresses among host machines. It requires much greater configuration effort, is likely not possible by way of DHCP. It also does not solve NDP table overflow attacks initiated by a compromised host on the LAN, which makes it a half-way solution. The knobs/features required to somewhat-mitigate the impact of an NDP table overflow attack are, at minimum: * keep NDP/ARP entry alive based on normal traffic flow, do not expire a host that is exchanging traffic + this is not the case with some major platforms, it surprised me to learn who does not do this + may require data plane changes on some boxes to inform control plane of on-going traffic from active addresses * have configurable per-interface limits for NDP/ARP resource consumption, to prevent attack on one interface/LAN from impacting all interfaces on a router + basically no one has this capability + typically requires only control plane modifications * have configurable minimum per-interface NDP/ARP resource reservation + typically requires only control plane modifications * have per-interface policer for NDP/ARP traffic to prevent control plane from becoming overwhelmed + because huge subnets may increase the frequency of scanning attacks, and breaking one interface by reaching a policer limit is much better than breaking the whole box if it runs out of CPU, or breaking NDP/ARP function on the whole box if whole-box policer is reached * learn new ARP/NDP entry when new transit traffic comes from a host on the LAN + even if NDP function is impared on the LAN due to on-going scan attack + again, per-interface limitations must be honored to protect whole box from breaking from one misconfigured / malicious LAN host * have sane defaults for all and allow all to be modified as-needed I am sure we can all agree that, as IPv6 deployment increases, many unimagined security issues may appear and be dealt with. This is one that a lot of smart people agree is a serious design flaw in any IPv6 network where /64 LANs are used, and yet, vendors are not doing anything about it. If customers don't express this concern to them, they won't do anything about it until it becomes a popular DoS attack. In addition, if you design your network around /64 LANs, and especially if you take misguided security-by-obscurity advice and randomize your host addresses so they can't be found in a practical time by scanning, you may have a very difficult time if the ultimate solution to this must be to change the typical subnet size from /64 to something that can fit within a practical NDP/ARP table. Deploying /64 networks because customers demand it and your competitors are doing it is understandable. Doing it because it's the standard is very stupid. Anyone doing that on, for example, SONET interfaces or point-to-point Ethernet VLANs in their infrastructure, is making a bad mistake. Doing it toward CE routers, the sort that have an IPv4 /30, is even more foolish; and many major ISPs already know this and are using small subnets such as /126 or /124. If you are still reading, but do not have any idea what I'm talking about, ask yourself these
Re: NIST IPv6 document
On Wed, Jan 5, 2011 at 9:39 AM, Iljitsch van Beijnum iljit...@muada.com wrote: that a lot of smart people agree is a serious design flaw in any IPv6 network where /64 LANs are used It's not a design flaw, it's an implementation flaw. The same one that's in ARP (or maybe RFC 894 wasn't published on april first by accident after all). And the internet managed to survive. It appears you want to have a semantic argument. I could grant that, and every point in my message would still stand. However, given that the necessary knobs to protect the network with /64 LANs do not exist on any platform today, vendors are not talking about whether or not they may in the future, and that no implementation with /64 LANs connected to the Internet, or any other routed network which may have malicious or compromised hosts, design flaw is correct. This is a much smaller issue with IPv4 ARP, because routers generally have very generous hardware ARP tables in comparison to the typical size of an IPv4 subnet. You seem to think the issue is generating NDP NS. While that is a part of the problem, even if a router can generate NS at an unlimited rate (say, by implementing it in hardware) it cannot store an unlimited number of entries. The failure modes of routers that have a full ARP or NDP table obviously vary, but it is never a good thing. In addition, the high-rate NS inquiries will be received by some or all of the hosts on the LAN, consuming their resources and potentially congesting the LAN. Further, if the router's NDP implementation depends on tracking the status of incomplete on-going inquiries, the available resource for this can very easily be used up, preventing the router from learning about new neighbors (or worse.) If it does not depend on that, and blindly learns any entry heard from the LAN, then its NDP table can be totally filled by any compromised / malicious host on the LAN, again, breaking the router. Either way is bad. This is a fundamentally different and much larger problem than those experienced with ARP precisely because the typical subnet size is now, quite literally, seventy-quadrillion times as large as the typical IPv4 subnet. On Wed, Jan 5, 2011 at 9:39 AM, Iljitsch van Beijnum iljit...@muada.com wrote: A (relatively) easy way to avoid this problem is to either use a stateful firewall that only allows internally initiated sessions, or a filter that lists only addresses that are known to be in use. It would certainly be nice to have a stateful firewall on every single LAN connection. Were there high-speed, stateful firewalls in 1994? Perhaps the IPng folks had this solution in mind, but left it out of the standards process. No doubt they all own stock in SonicWall and are eagerly awaiting the day when Anonymous takes down a major ISP every day with a simple attack that has been known to exist, but not addressed, for many years. You must also realize that the stateful firewall has the same problems as the router. It must include a list of allocated IPv6 addresses on each subnet in order to be able to ignore other traffic. While this can certainly be accomplished, it would be much easier to simply list those addresses in the router, which would avoid the expense of any product typically called a stateful firewall. In either case, you are now maintaining a list of valid addresses for every subnet on the router, and disabling NDP for any other addresses. I agree with you, this knob should be offered by vendors in addition to my list of possible vendor solutions. On Wed, Jan 5, 2011 at 9:39 AM, Iljitsch van Beijnum iljit...@muada.com wrote: Sparse subnets in IPv6 are a feature, not a bug. They're not going to go away. I do not conceptually disagree with sparse subnets. With the equipment limitations of today, they are a plan to fail. Let's hope that all vendors catch up to this before malicious people/groups. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: AltDB?
On Wed, Jan 5, 2011 at 11:26 AM, Jon Lewis jle...@lewis.org wrote: Anyone here use AltDB? It seems their servers have been down for two days. Can anyone from Level3 say how this will impact customer BGP filters. Will L3 keep working with the last data sync they got from altdb? I'm guessing Since Level3 updates their prefix-lists at least daily, and integrates new ALTDB updates at least daily, and the ALTDB has been down for over a day, obviously it will not affect your Level3 prefix-lists in the near-term. If Level3 decided to stop honoring ALTDB objects, say, because ALTDB was never fixed, I imagine you would find it necessary to re-publish your objects or Level3 would stop honoring your routes. I'd been thinking about moving from altdb to ARIN's but hadn't had sufficient motivation. I emailed ARIN yesterday to ask if their IRR database has any authentication support (other than mail-from) yet. I haven't seen any reply from ARIN yet, but my guess is they still have no useful authentication mechanism. I would rather depend on an IRR database that can't process updates for a few days per year, than use one where a malicious party could alter or erase all of my objects at any time. I would like to note that RADB had route6: support in about 2004 or so, if my memory serves me; while the ARIN database did not accept route6 objects until about a year ago. So it is not exactly a high priority for ARIN. Note also that Level3 has an IRR database, so you could use theirs if you want to. I don't prefer to use a transit provider database if I can use a neutral one, but sometimes I would rather not pay the (entirely reasonable) fee for the MERIT RADB. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Wed, Jan 5, 2011 at 12:04 PM, Joel Jaeggli joe...@bogus.com wrote: no it isn't, if you've ever had your juniper router become unavailable because the arp policer caused it to start ignoring updates, or seen systems become unavailable due to an arp storm you'd know that you can abuse arp on a rather small subnet. These conditions can only be triggered by malicious hosts on the LAN. With IPv6, it can be triggered by scanning attacks originated from the Internet. No misconfiguration or compromised machine on your network is necessary. This is why it is a fundamentally different, and much larger, problem. Since you seem confused about the basic nature of this issue, I will explain it for you in more detail: IPv4) I can scan your v4 subnet, let's say it's a /24, and your router might send 250 ARP requests and may even add 250 incomplete entries to its ARP table. This is not a disaster for that LAN, or any others. No big deal. I can also intentionally send a large amount of traffic to unused v4 IPs on the LAN, which will be handled as unknown-unicast and sent to all hosts on the LAN via broadcasting, but many boxes already have knobs for this, as do many switches. Not good, but also does not affect any other interfaces on the router. IPv6) I can scan your v6 /64 subnet, and your router will have to send out NDP NS for every host I scan. If it requires incomplete entries in its table, I will use them all up, and NDP learning will be broken. Typically, this breaks not just on that interface, but on the entire router. This is much worse than the v4/ARP sitation. I trust you will understand the depth of this problem once you realize that no device has enough memory to prevent these attacks without knobs that make various compromises available via configuration. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Wed, Jan 5, 2011 at 12:26 PM, Phil Regnauld regna...@nsrc.org wrote: Jeff Wheeler (jsw) writes: Not good, but also does not affect any other interfaces on the router. You're assuming that all routing devices have per-interface ARP tables. No, Phil, I am assuming that the routing device has a larger ARP table than 250 entries. To be more correct, I am assuming that the routing device has a large enough ARP table that any one subnet could go from 0 ARP entries to 100% ARP entries without using up all the remaining ARP resources on the box. This is usually true. Further, routing devices usually have enough ARP table space that every subnet attached to them could be 100% full of active ARP entries without using up all the ARP resources. This is also often true. To give some figures, a Cisco 3750 pizza box layer-3 switch has room for several thousand ARP entries, and I have several with 3000 - 5000 active ARPs. Most people probably do not have more than a /20 worth of subnets for ARPing to a pizza box switch like this, but it does basically work. As we all know, a /64 is a lot more than a few thousand possible addresses. It is more addresses than anyone can store in memory, so state information for incomplete can't be tracked in software without creating problems there. Being fully stateless for new neighbor learning is possible and desirable, but a malicious host on the LAN can badly impact the router. This is why per-interface knobs are badly needed. The largest current routing devices have room for about 100,000 ARP/NDP entries, which can be used up in a fraction of a second with a gigabit of malicious traffic flow. What happens after that is the problem, and we need to tell our vendors what knobs we want so we can choose our own failure mode and limit damage to one interface/LAN. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Wed, Jan 5, 2011 at 1:02 PM, TJ trej...@gmail.com wrote: Many would argue that the version of IP is irrelevant, if you are permitting external hosts the ability to scan your internal network in an unrestricted fashion (no stateful filtering or rate limiting) you have already lost, you How do you propose to rate-limit this scanning traffic? More router knobs are needed. This also does not solve problems with malicious hosts on the LAN. A stateful firewall on every router interface has been suggested already on this thread. It is unrealistic. Even granting that, for the sake of argument - it seems like it would not be hard for $vendor to have some sort of emergency garbage collection routines within their NDP implementations ... ? How do you propose the router know what entries are garbage and which are needed? Eliminating active, good entries to allow for more churn would make the problem much worse, not better. -- Jeff S Wheeler j...@inconcepts.biz +1-212-981-0607 Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Wed, Jan 5, 2011 at 8:57 PM, Joe Greco jgr...@ns.sol.net wrote: This is a much smaller issue with IPv4 ARP, because routers generally have very generous hardware ARP tables in comparison to the typical size of an IPv4 subnet. no it isn't, if you've ever had your juniper router become unavailable because the arp policer caused it to start ignoring updates, or seen systems become unavailable due to an arp storm you'd know that you can abuse arp on a rather small subnet. It may also be worth noting that typical size of an IPv4 subnet is a bit of a red herring; a v4 router that's responsible for /16 of directly attached /24's is still able to run into some serious issues. It is uncommon for publicly-addressed LANs to be this large. The reason is simple: relatively few sites still have such an excess of IPv4 addresses that they can use them in such a sparsely-populated manner. Those that do have had twenty years of operational experience with generation after generation of hardware and software, and they have had every opportunity to fully understand the problem (or redesign the relevant portion of their network.) In addition, there is not (any longer) a standard, and a group of mindless zealots, telling the world that at /16 on your LAN is the only right way to do it. This is, in fact, the case with IPv6 deployments, and will drive what customers demand. To understand the problem, you must first realize that myopic standards-bodies have created it, and either the standards must change, operators must explain to their customers why they are not following the standards, or equipment vendors must provide additional knobs to provide a mitigation mechanism that is an acceptable compromise. Do the advantages of sparse subnets out-weigh the known security detriments, even if good compromise-mechanisms are provided by equipment vendors? Security by obscurity is an oft-touted advantage of IPv6 sparse subnets. We all know that anyone with a paypal account can buy a list of a few hundred million email addresses for next to nothing. How long until that is the case with lists of recently-active IPv6 hosts? What portion of attack vectors really depend on scanning hosts that aren't easily found in the DNS, as opposed to vectors depending on a browser click, email attachment, or by simply hammering away at www.*.com with common PHP script vulnerabilities? How many people think that massively-sparse-subnets are going to save them money? Where will these cost-efficiencies come from? Why can't you gain that advantage by provisioning, say, 10 times as large a subnet as you think you need, instead of seventy-quadrillion times as large? Is anyone really going to put their Windows Updates off and save money because they are comfortable that their hosts can't be found by random scanning? Is stateless auto-configuration that big a win vs DHCP? Yes, I should have participated in the process in the 1990s. However, just because the bed is already made doesn't mean I am willing to lay my customers in it. These problems can still be fixed before IPv6 is ubiquitous and mission-critical. The easiest fix is to reset the /64 mentality which standards-zealots are clinging to. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
NIST IPv6 document
On Thu, Jan 6, 2011 at 12:17 AM, Joe Greco jgr...@ns.sol.net wrote: However, that's not the only potential use! A client that initiates each new outbound connection from a different IP address is doing something Really Good. No, Joe, it is not doing anything Good. This would require the software being written to make such random address selection, add more entries to the router's NDP table, and it would DoS the box's own router if an outbound scan were initiated from the host machine. Again, you totally fail to understand the problem. I should just attach a facepalm graphic to my reply and stop bothering with your idiocy, but it is important that as many people as possible understand these issues. Every additional person who is expressing concern to their vendors brings us closer to a solution. In addition, if the machine can choose another random IPv6 address in the /64 for each new outgoing connection, it could just as easily source all its outgoing connections from ... a single IPv6 address that does not accept any incoming connections. I will grant that this does not prevent magic packet attacks against bugs in the machine in kernel space, and that the machine could be the recipient of a DoS attack; but your proposed solution has no advantage in the case of, say, SQL Slammer 6, or other attacks exploiting vulnerabilities in user-space. On Thu, Jan 6, 2011 at 12:25 AM, Owen DeLong o...@delong.com wrote: Is there any reason we really need to care what size other people use for their Point to Point links? Personally, I think /64 works just fine. I won't criticize anyone for using it. It's what I choose to use. You should care what size you choose, Owen. I would if I was your customer. There are several reasons why. If you configure a /64 on a point-to-point interface e.g. SONET, some current platforms will bounce any scan packets back and forth between the two hosts until the TTL expires, which means an attacker can saturate the point-to-point link with two orders of magnitude less attack traffic than the link capacity. This is very bad. Also, if the link is a LAN, you are vulnerable to the ARP/NDP problem. This may be a per-interface issue on your box, or it may be a global one. In the per-interface case, most likely, the failure mode will be okay; but some vendors (including Juniper?) will eventually expire the ARP/NDP entry even though there is active traffic, and may be unable to re-learn it when needed due to the attack, which would break the link between routers. On the other hand, if it's a global issue, it will break NDP on the entire router, affecting all interfaces by whatever the platform's failure mode is. Obviously, that is worse. This is why RFC compliant is wrong, Owen. That a learned, experienced person such as yourself has no clue what the underlying problem is exemplifies my point: much, much more noise needs to be made about this issue. A lot of smart people have known about it for years but nothing is being done. Anyone can choose not to use /64 on their point-to-point links with no consequence to their business. Customers aren't going to choose another vendor because they are likely never even aware of it. The operational problem is that customers will want it on the PE/CE LANs, hosting center LANs, and so on; and right now, if you tell them we can't do that, they have trouble believing you because the standard says otherwise, and a lot of other networks are currently doing it (because they think they don't have a choice other than to lose potential customers.) However, if you are running /64 instead of something like /124 on your VLANs or SONET circuits between routers, you should change this practice as soon as practical. Not doing so once you have been educated about this problem because it is the standard is very stupid. Pretty much every major transit network has figured this out and are doing it on their interfaces to BGP customers, too, either optionally, by default, or as the only choice. The industry standard must change to be non-hostile toward choosing a smaller subnet size than /64 to give coverage to those of us who do understand what we are doing as we roll out IPv6 to customers. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Thu, Jan 6, 2011 at 12:54 AM, Joe Greco jgr...@ns.sol.net wrote: I'm starting off with the assumption that knowledge of the host address *might* be something of value. If it isn't, no harm done. If it is, and the address becomes virtually impossible to find, then we've just defeated an attack, and it's hard to see that as anything but positive. I'm starting off with the assumption that the layer-3 network needs to function for the host machines to be useful. Your position is to just hand any attacker an off switch and let them disable any (LAN | router, depending on router failure mode) for which they know the subnet exists, whether or not they know any of its host addresses. This is a little like spending money on man-traps and security guards, but running all your fiber through obvious ducts in a public parking garage. It may be hard to compromise the hosts, but taking them offline is trivial. On Thu, Jan 6, 2011 at 1:01 AM, Kevin Oberman ober...@es.net wrote: I am amazed at the number of folks who seem to think that there is time to change IPv6 is ANY significant way. Indeed, the ship has failed. If you r network is not well along in getting ready for IPv6, you are probably well on you way to being out of business. There are many things that can change very easily. Vendors can add knobs, subnet size can get smaller (it works just fine today, it just isn't standard), and so on. A TCP session today looks a lot different than it did in the mid-90s. Now we have things like SYN cookie, window scaling, we even went through the hurry up and configure TCP MD5 on your BGP just in case. Fixing this problem by deploying subnets as a /120 instead of a /64 is a lot easier than any of those changes to TCP, which all required operating system modifications on one or both sides. How many networks honor ICMP route-record, source routing, or make productive use of redirects (if they have not outright disabled it?) How many networks decided to block all ICMP traffic because some clueless employee told them it was smart? CIDR routing? Do you recall that the TTL field in IP headers was originally not a remaining-hops-count, but actually, a value in seconds (hence Time To Live)? IPv4, and the things built on top of it, have evolved tremendously, some, all the way to the host network. A lot of this evolution took place before it was common to conduct a credit card transaction over the Internet, at a time when it really was not mission-critical for most operators. IPv6 is still not there, but I agree, we are rapidly approaching that time, and much more than 90% of IPv4 networks have a lot of work to do. It would be good to see LANs smaller than /64 accepted sometime before IPv6 does become widely-deployed to end users. Or some other practical solution to the problems of huge subnet sizes, whatever those solutions may be. My guess is there may be other, very significant, challenges to having huge LAN subnets. This is one we actually know about, but are choosing not to solve. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: NIST IPv6 document
On Tue, Jan 4, 2011 at 11:35 PM, Kevin Oberman ober...@es.net wrote: The PDF is available at: I notice that this document, in its nearly 200 pages, makes only casual mention of ARP/NDP table overflow attacks, which may be among the first real DoS challenges production IPv6 networks, and equipment vendors, have to resolve. Some platforms have far worse failure modes than others when subjected to such an attack, and information on this subject is not widely-available. Unless operators press their vendors for information, and more knobs, to deal with this problem, we may all be waiting for some group like Anonymous to take advantage of this vulnerability in IPv6 networks with large /64 subnets configured on LANs; at which point we may all find ourselves scrambling to request knobs, or worse, redesigning and renumbering our LANs. RFC5157 does not touch on this topic at all, and that is the sole reference I see in the NIST publication to scanning attacks. I continue to believe that a heck of a lot of folks are missing the boat on this issue, including some major equipment vendors. It has been pointed out to me that I should have been more vocal when IPv6 was still called IPng, but in 16 years, there has been nothing done about this problem other than water-cooler talk. I suspect that will continue to be the case until those of us who have configured our networks carefully are having a laugh at the networks who haven't. However, until that time, it's also been pointed out to me that customers will expect /64 LANs, and not offering it may put networks at a competitive disadvantage. Vendor solutions are needed before scanning IPv6 LANs becomes a popular way to inconvenience (at best) or disable (at worst) service providers and their customers. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: IPv6 BGP table size comparisons
On Wed, Dec 22, 2010 at 2:24 AM, Pekka Savola pek...@netcore.fi wrote: 'Maximum Prefix Length' may be an over-simplifying metric. FWIW, we're certainly not a major transit provider, but we do allow /48 in the designated PI ranges but not in the PA ranges. So the question is not necessarily just about the prefix length used because it might vary by the prefix. I know it is an over-simplification. If someone wishes to edit the page to provide more specific details about the route filtering policy for a given transit network, Wikipedia is pretty easy to edit. Hopefully they would provide a citation/link to the policy page for the NSP as well. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: IPv6 BGP table size comparisons
I could not find this information on any Wikis, but this is the sort of thing that would be nice to be able to find out without posting on the list or asking around (obviously.) I have quickly made a couple of entries with simple enough formatting that anyone can go onto Wikipedia, click Edit, and add what they know. This is sure to become a frequently asked question before the answer is always yes given that some major transit-free networks have no functional IPv6 capability of any kind. http://en.wikipedia.org/wiki/Comparison_of_IPv6_support_by_major_transit_providers -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Some truth about Comcast - WikiLeaks style
On Sun, Dec 19, 2010 at 8:48 PM, Richard A Steenbergen r...@e-gerbil.net wrote: Running a wire to everyone's house is a natural monopoly. It just doesn't make sense, financially or technically, to try and manage 50 different companies all trying to install 50 different wires into every house just to have competition at the IP layer. It also wouldn't make What no one has mentioned thus far is that CLECs really are able to install their own facilities to homes and businesses if they decide that is a good way to invest their finite resources. This is why we see several options for local loops in the business district of every sizable city, as well as in many newly-developed areas such as industrial parks. These infrastructure builds are expensive, the CLECs had limited logistical capabilities and could only manage so many projects at once, and obviously, they focused their efforts on the parts of town where return-on-investment was likely to be highest. Businesses often do have several good choices for voice, data, Internet, and so on. Cogent is an example of an essentially Internet-only service having some degree of success at this without even offering voice, or initially even transport, products. The reason we will not see competitive facility builds to residences is they have a very long ROI scale. Everything in the traditional telecommunications world did. Many POTS customers still pay a fee for DTMF or touch tone dialing, because when their phone company invested in new cards and software to support DTMF signalling, they passed those expenses on to consumers. These upgrades cost on the order of a thousand dollars per phone line, but consumers could get the benefit of DTMF by paying a couple dollars per month. See also: call waiting, caller ID, and so on. I don't know about you, but I was still using an ATDP dialing string until cable and DSL became available to me at home (in about 2002) because I did not want to pay the extra fee for touch tone dialing or other features I didn't need on a dedicated modem line. ;) We see examples of more choice available to business consumers than residences, due to economies of scale, every day. A business, apartment community, or neighborhood association can choose from multiple dumpster-tip services for trash collection. Most residents do not have enough trash volume to justify a bulky dumpster, so their only practical choice is whatever curb-side trash collection company has an agreement with their local government. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: potential new and different architectural approach to solve the Comcast - L3 dispute
On Fri, Dec 17, 2010 at 12:15 PM, Benson Schliesser bens...@queuefull.net wrote: I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The modest investment is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either. That is a reasonable guess, but Level3's FCC filing yesterday spells out with certainty that Level3 did offer to cold potato traffic onto Comcast (it does not mention the technical means e.g. MED honoring, CDN smarts, or otherwise) and that Comcast refused. I agree that the proposed Comcast solution may not be truly new but instead unusual, but unless Backdoor Santa tells us what they really have in mind, I suppose we won't know. If I were Comcast, I would want to move the significant cost of detailed netflow collection and analysis infrastructure onto backbone providers by wrapping that accounting mechanism up into my settlement agreements with peers, as well as the expense of a cost-ineffective network, and demand that Level3 and Comcast really calculate how much each network spends on each bit, and share in that cost. In theory, this is what happens when an ILEC opens a rate case with its state regulator; and it is how settlements for POTS calls work (at a very basic level.) Actually, if I were Comcast, I would focus on running my business more efficiently, as Level3 has thrown down the gauntlet with the FCC and requested that the FCC dictate to Comcast specifically, and explicitly all other broadband access providers, how they will interconnect with peers and transit suppliers. Level3 must think that their business would be better off with regulatory oversight of peering, or they would not have taken this action. Comcast should realize that, of the three potential motives for their recent actions I have previously outlined, #1 and #3 are not just highly unlikely, but would be practically impossible in a regulated environment. As such, they should further realize that their peering committee is driven by motive #2, ego, and find the best way to change their position without losing too much credibility. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: potential new and different architectural approach to solve the Comcast - L3 dispute
On Fri, Dec 17, 2010 at 12:48 PM, Richard A Steenbergen r...@e-gerbil.net wrote: advertising MEDs, or by sending inconsistent routes. The fact that the existing Level3/Comcast routing DOESN'T make Level 3 haul all of the bits to the best exit mean it's highly likely that Comcast agreeing to haul the bits was part of their commercial transit agreement, probably in exchange for lower transit prices. It's worth asking why Comcast did not accept Level3's suggestion that they use MED as a face-saving maneuver, which would have allowed both sides to declare victory. A) Comcast may already have the contractual right to use MED but chooses not to. I agree with you that this is unlikely, not for pure reasons of economics, but because Comcast has some of the same set of motives not to send MED to their transit provider as every other network: prefix aggregation, quality control, and ego. I'll discount geography, marketing, and inability to calculate useful MED values. For argument's sake, let's say they currently can start sending MEDs to Level3 whenever they want. This being the case, Level3's offer would have amounted to Level3 telling Comcast upper management that Comcast's engineering people are leaving a huge amount of money on the table, that Level3 is far more cost-effective at running its long-haul network than Comcast, and that they should leave the big networking to the big boys. Comcast management could either react badly to this, or go back to their network folks and ask why they can't be as cost-effective as Level3. B) Comcast may not be able to use MED today. In this case, management may be asking themselves why. An essentially similar scenario can play out; they can either react badly to Level3, or ask their own staff why they are wasting money. C) Comcast doesn't care about MED or the actual cost of doing business. They are boldly moving towards a future that is opposite the one net neutrality folks advocate, one that looks like my Comcast Motive #3. D) Comcast does not think that beginning to use MED (whether currently enabled or not) is enough to satisfy the federal regulators and legislators who are now taking interest in this game of interconnection brinkmanship, involving 17 million households, between a major IP carrier delivering traffic from everyone including a household name like Netflix, and a major cable company that is waiting for government approval to purchase NBC. They feel they must demand something very concrete to demonstrate that they are looking out for consumers' best interest, which means they must make Level3 and/or Netflix look like the bad guy. E) Comcast thinks that a system of accounting for the cost of bearing traffic and dividing it among the involved parties will actually be good for their business, because they can over-build their infrastructure as much as they like, perhaps even improving quality for end-users, and only have to pay for about half of it. The cost of being inefficient, stupid, or committing purchasing or forecasting errors drops by half. This looks very much like my Comcast Motive #1. E1) Comcast may also know a thing or two about Hollywood Accounting. If you do not understand this reference, simply look it up on Wikipedia. It suffices to say that cost/revenue sharing agreements of this nature can be manipulated in gross ways to the advantage of the party doing the bulk of the book-keeping. F) Management has the same case of ego-driven decision-making that their technical staff have demonstrated. I find this unlikely but still possible. We all know this has been the case at the CEO level in some major interconnection disputes of the past. I believe this outlines the reasonable scenarios for Comcast avoiding a face-saving maneuver with Level3. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Some truth about Comcast - WikiLeaks style
On Thu, Dec 16, 2010 at 12:15 PM, Dave Temkin dav...@gmail.com wrote: I disagree. Even at $1/Mbit and 6Tbit of traffic (they do more), that's still $72M/year in revenue that they weren't recognizing before. Given that that traffic was actually *costing* them money to absorb before, turning the balance and making that kind of money would be very favorably looked upon in Yeah, because it makes a lot of sense to fuck with a billion dollar a month revenue stream so you can extract a few million dollars more per month from IP carriers. This definitely makes more sense than, say, running the billion dollar a month side a little more efficiently. You need to understand the scale of comcast's expenses and revenue on the access and transport side of their business, in order to have a remotely intelligent opinion about whether or not they are doing anything smart with the peering/transit side, in these conditions. If you really think it's a good idea to attract the attention of government regulators, newspapers, customers, and every major ISP by making a lot of noise over something that might allow you to make 0.5% more money off a product where you could probably save an order of magnitude more money through any number of ignored efficiencies within the organization, I would love for you to post that. I suspect that most folks who are of the opinion that Comcast is motivated by anything but the three things I mentioned have not clearly considered the proportionately small benefit they could gain from selling access to their network at anything approaching a nominal fee. It must be either 1) very high per-Mb price; 2) ego and stupidity; 3) greed of such magnitude that it would make Gordon Gecko proud. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Some truth about Comcast - WikiLeaks style
On Thu, Dec 16, 2010 at 1:53 PM, Dave Temkin dav...@gmail.com wrote: I do. And yes, they are happy to fuck with a billion dollar a month revenue stream (that happens to be low margin) in order to set a precedent so that when traffic is 60Tbit instead of 6Tbit, across the *same* customer We disagree on this point. I do not think anyone knowledgeable at Comcast realistically believes they will be able to charge a business-relevant amount of money for access to their customers. I think regulators would first but the brakes on our whole industry. Cable Internet is far from low-margin; in fact, the cable company in my area, an order of magnitude smaller than Comcast, generates an order of magnitude more profit from IP than from television. What I do think, and what people on this list who engage in peering discussions with Comcast cannot say for fear of reprisal, is that the peering folks at Comcast are driven entirely by ego, and they lack the big picture decision-making capacity of business people making strategy decisions. They have upper management convinced that becoming settlement-free is a golden goose. The peering folks would be wise to reconsider their positions before upper management realizes that their ego-driven staff are risking the golden goose they already have, their captive audience, for little gain. After all, if they can't manage to run their IP and transport network more cost-effectively than they do today, they will never be able to compete as a transit network, and the golden goose they are chasing will never lay any eggs. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Alacarte Cable and Geeks
On Fri, Dec 17, 2010 at 12:26 AM, Jay Ashworth j...@baylink.com wrote: the 80s when that practice got started -- having to account for each individual subscriber pushed the complexity up, in much the same way that flat rate telecom services are popular equally because customers prefer them, and because the *cost of keeping track* becomes delta. Having personally and solely designed and written a toll billing system from scratch that directly exchanged billing and settlement data (and end-user data) with hundreds of ILECs, I can tell you a number of things I learned: 1) billing is only as hard as you (or your vendor) make it 2) if your company can't figure out how to bill for a new product or service, blame the billing people, not the product 3) keeping up with taxes and fees consume a lot more resources than calculating the net bills themselves; so adding products is really trivial compared to dealing with every pissant local government that decides to apply a different taxing method to your HBO (or your telephone calls) This is not to say the folks that handle billing at cable companies are equally capable, but if they had legitimate competitors, they would figure out how to run many parts of their businesses more efficiently. Imagine if Wal-Mart was the only game in town that had bar code readers at the cash registers, and every other grocery chain had to look up every item and punch in the price to check you out. Other stores would quickly improve their technology or find themselves out of business. 2) New networks prefer it, and the fact that it happens makes the creation of new cable networks practical -- you don't have to go around and sell your idea to people retail; you sell it to CATV systems (well, My understanding is that networks/media giants like it because they can force cable companies to carry 11 irrelevant channels to get the Disney Channel that your kids want. Would enough people really ask for G4TV to make producing and syndicating shows for that channel cost-effective? I don't know the answer, but my suspicion is that people who really just want CSN, E!, or the Golf Channel are subsidizing G4 viewers. I wanted BBCA a few years ago, but my cable provider required that I buy 30 other channels I did not want or had never even heard of to get BBCA, so I didn't subscribe to it. I do not know if a la carte channel selection would be good for me, as a consumer, or not. I do think the reasons the industry does not want to offer that to end-users are disingenuous. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: peering, derivatives, and big brother
Invisible Hand Networks was really meant to be a spot market. The same problem exists with bandwidth spot markets that always has existed, the cost of ports to maintain sufficient capacity to the exchange, and the lack of critical mass, meaning that the spot bandwidth is either pretty expensive, or there is not enough capacity for any serious application. Certainly, no spot bandwidth market currently in existence can compete with even mid-sized CDNs; and I do not believe that will ever change. The IHN folks were also disadvantaged because they seemed to know a lot about economics, but basically nothing about networks. So their technology was neat from a reporting perspective, but the actual functioning their exchange fabric was/is a disaster. I do not know if they are still in business or if they are still constrained by the flawed design they had in place several years ago. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts On Wed, Dec 15, 2010 at 2:52 PM, Ryan Finnesey ryan.finne...@harrierinvestments.com wrote: I remember 5 years ago a company called Invisible Hand Networks that tried something like that. Cheers Ryan -Original Message- From: Laurent GUERBY [mailto:laur...@guerby.net] Sent: Monday, December 13, 2010 3:07 PM To: George Bonser Cc: nanog@nanog.org Subject: Re: peering, derivatives, and big brother On Sun, 2010-12-12 at 19:36 -0800, George Bonser wrote: (...) The financial derivatives market isn't, in my opinion, a good analogy of the peering market. A data packet is perishable and must be moved quickly. The destination network wants the packet in order to keep their customer happy and the originating network wants to get it to that customer as quickly and cheaply as possible. The proliferation of these peering points means that today there is more traffic going directly from content network to eyeball network. To use a different analogy, it is almost like the market is going to a series of farmer's markets rather than supermarkets in the distribution channel. Sure, there are still the supermarkets out there, but increasingly they are selling their store brand by becoming content hosting networks themselves. (...) Hi, The electricity spot market is close to your definition of perishable: http://en.wikipedia.org/wiki/Electricity_market It has a derivative market, google for electricity derivatives will give you some papers and models. I'm pretty sure electricity and bandwidth share some patterns. Now who wants to be the Enron of the bandwidth market? :) Sincerely, Laurent http://guerby.org/blog
Re: Some truth about Comcast - WikiLeaks style
On Wed, Dec 15, 2010 at 5:47 PM, Adam Rothschild asr+na...@latency.net wrote: I don't see how this point, however valid, should factor into the discussion. Missing from this thread is that Comcast's topology and economics for hauling bits between a neutral collocation facility and broadband subscriber are the _same_ whether they ingest traffic by way of a settlement-free peer, customer, or paid transit connection. Given that transit must be an incredibly small portion of Comcast's cost to provide IP service to its customers, I think there are only three possible reasons why Comcast would focus so much energy on congesting transit to force content networks to purchase connectivity for them, rather than upgrade transit or engage in more peering: 1) Comcast believes they can exact a great deal of revenue from content networks. For this to be comparable to their captive customers, per-megabit rates must be reminiscent of pre-Level3 days, when $30/Mb was a bargain. This would spell bad news for Netflix. Of course, since cable companies typically must pay network affiliates and media companies great sums for television programming packages, it is in direct opposition to the TV content/delivery model. It would be hard to argue both sides if both businesses were faced with like-minded regulators. 2) Comcast is making its engineering decisions in an ego-driven manner, with little or no practical basis for their peering or transit purchasing strategy. 3) Comcast is hoping the phrase net neutrality becomes a thing of the past, and that, at some point in the future, they will be free to block or QoS down anyone they please, including content networks, search engines, or MP3 stores that compete with their own offerings. I bet Comcast would love to have a few cents off every iTunes purchase through their network, a handling charge for every amazon.com transaction, or to coerce a million Netflix subscribers into a Comcast-owned service. This is as good a way as any for Comcast to argue their side to potential regulators. In any case, the net neutrality side gains credibility anytime a media company can be made to look like they are constraining users' choices by exacting a price from content providers. There has been talk of regulating Internet peering on this list since DIGEX disconnected from ANS, if not before. Reasons in favor of doing it continue to become easier for a lay-person to understand. In my state, there is a law against walking down the street with an ice cream cone in your pocket. I don't know the origin of that law, but I strongly suspect some person did it enough times, for a dumb enough reason, to attract legislative interest. Comcast should keep that in mind before engaging in further peering brinkmanship. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
peering, derivatives, and big brother
A read through this New York Times article on derivatives clearing, and the exclusivity that big banks seek to maintain, would look very much like an article on large-scale peering, to someone who is not expert in both topics. The transit-free club and the derivatives dealers club may have other similarities in the future, and it's worth watching how further government regulation develops in this area. It may lead to insight into how government might eventually regulate ISPs seeking to become settlement-free. “It appears that the membership criteria were set so that a certain group of market participants could meet that, and everyone else would have to jump through hoops,” Mr. Katz said. http://www.nytimes.com/2010/12/12/business/12advantage.html?pagewanted=1_r=1src=busln -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Videotron contact
Could someone from Videotron contact me off-list? -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Start accepting longer prefixes as IPv4 depletes?
How many networks already leak numerous unnecessary /24s to their transit providers, who accept them (not having been asked to do anything else), and contribute to table bloat? Quite a lot of networks do this. Imagine if there are many possible inter-domain routes that are being filtered by transit networks, because their customers accidentally announce some number of /25-/32 networks to them. These do not affect us today; but I would hate to see all those accidental announcements suddenly appear in my routing table; or for my transit providers to have the bear the expense of dealing with them. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: The scale of streaming video on the Internet.
On Thu, Dec 2, 2010 at 3:38 PM, Seth Mattinen se...@rollernet.us wrote: On 12/2/10 12:28 PM, Owen DeLong wrote: You are assuming the absence of any of the following optimizations: 1. Multicast Multicast is great for simulating old school broadcasting, but I don't see how it can apply to Netflix/Amazon style demand streaming where I do. Let's assume that there is a multicast future where it's being legitimately used for live television, and whatever else. The same mcast infrastructure will be utilized by Amazon.com to stream popular titles (can you say New Releases) onto users' devices. You may be unicast for the first few minutes of the movie (if you really want to start watching immediately) and change over to a multicast-distributed stream once you have caught up to an in-progress stream. If Netflix had licensing agreements which made it possible for their users to store movies on their local device, this would work even better for Netflix, because of the queue and watch later nature of their site and users. I have a couple dozen movies in my instant queue. It may be weeks before I watch them all. The most popular movies can be multicast, and my DVR can listen to the stream when it comes on, store it, and wait for me to watch it. I am sure Amazon and Netflix have both thought of this already (if not, they need to hire new people who still remember how pay-per-view worked on C-band satellite) and are hoping multicast will one-day come along and massively reduce their bandwidth consumption on the most popular titles. I am also certain the cable companies have thought of it, and added it to the long list of reasons they will never offer Internet multicast, or at least, not until a competitor pops up and does it in such a way that customers understand it's a feature they aren't getting. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
TWT - Comcast congestion
On Tue, Nov 30, 2010 at 9:12 PM, Richard A Steenbergen r...@e-gerbil.net wrote: uncongested access. This is the kind of action that virtually BEGS for government involvement, which will probably end badly for all networks. This depends on the eventual regulatory mechanism and the goals it intends to promote. Everyone in our industry has been aware that security mechanisms related to BGP are needed, but after major incidents making it into the news regularly for ten years, little progress has been made. A regulator putting the hammer down might be a driving force to solve some of our basically solvable problems that no one is willing to spend any time or money on. Additionally, it is easy to make the argument that reduced interconnection cost for end-user ISPs would never motivate any innovation. If any network with 1000 DSL users could connect to the closest PAIX (in every NFL city, of course) and gain access to all the big players for nothing but the cost of transport, it would not significantly reduce their cost to serve their customers. The DSLAMs, tech support monkeys, transport, idiotic implementation choices, etc. cost an order of magnitude more than transit. No regulator is going to believe that eliminating the cost of transit will encourage more broadband deployment, higher broadband speeds, or new inventions that tax the network more heavily. On the other hand, it is very easy for regulators to imagine that, if Youtube had to bear the whole cost of moving bits from them to the end-user, and broadband access was free for anyone with a house and mailbox, developing new applications would be much more expensive and happen less frequently. I think eyeball networks had better start demonstrating how they are innovating new things that benefit the public, and working hard to run their networks and businesses efficiently, before the regulation gauntlet is thrown down. Otherwise, they will be on the losing end. In either case, I don't think it automatically must be bad for all networks, and everyone except those eyeball networks should hope it turns out to be good for the public, increasing consumer choice and bringing new forms of information and entertainment into their homes. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Level 3 Communications Issues Statement Concerning Comcast's Actions
On Mon, Nov 29, 2010 at 11:20 PM, Leo Bicknell bickn...@ufp.org wrote: I will be the first to advocate the government use minimal to no regulation where there is active competition and consumer choice, and thus folks can vote with their dollars. Broadband in the US is not in that boat. Too many consumers have a choice of a single provider. The vast majority of the rest have the choice of two providers. We make these monopoly or I believe regulation of peering among the largest networks in the U.S. is a question of when and how, not if. The more these incidents make it into the news and attract the attention of public policy-makers, the closer that when may become. Comcast is either very clever, or very stupid, for timing this in such a way that it has been spun into an issue of who is streaming what into their customers' living rooms. -- Jeff S Wheeler j...@inconcepts.biz
