Re: The Making of a Router
On Sat, Dec 28, 2013 at 8:09 AM, sten rulz wrote: > Hello Baldur, > > Your design regarding proxy arp for every VLAN might hit some issues. If > you look at the nanog history you will find people having issues with proxy > arp for large number of VLANs, what is your requirement for proxy arp? > Doing something at the access switch will most likely be better for you > such as PVLAN or Brocade IP follow ve statement. If you are planning to put > clients on the same subnet what are you planning to put in place to limit > client stealing each other’s IPs? Only a few Brocade devices support the > ARP ACLs rules which are a really nice feature, IP Source Guard works > reasonable if using a DHCP server otherwise you need to specify the MAC > address. Some other brand switches support filtering the ARP packets per > access port. > This is a complex question that depends entirely on the capabilities of the equipment I can get. I was considering an OpenFlow solution, where this is easy: I would make rules that would only forward traffic with correct source IP from each VLAN. If the user tries something funny, nothing happens and his traffic is just dropped. But I am bit let down on the capabilities of current OpenFlow switches. Most only support OpenFlow 1.0 which is simply not good enough. That has no IPv6 support, which naturally is a requirement. I know about the HP offerings, but they only support 4k rules in hardware, which is a far cry from being enough. There is NoviFlow who are still working on getting me a quote. If they can give me a competitive price I might still consider OpenFlow. The problem is this: A conventional approach assigns a full IPv4 subnet to each user. This uses a minimum of 4 addresses of each user. I currently have to pay somewhere between $10 and $20 for each address and this will only become more expensive in the future. The users each have a unique VLAN (Q-in-Q). The question is, what do I put on those VLANs, if I do not want to put a full IPv4 subnet on each? My own answer to that is to have the users share a larger subnet, for example I could have a full class C sized subnet shared between 253 users/VLANs. To allow these users to communicate with each other, and so they can communicate with the default gateway IP, I will need proxy arp. And in a non-OpenFlow solution, also the associated security functions such as DHCP-snooping to prevent hijacking of IP addresses. Which devices can solve this task? To me the work seems quite simple. For outbound packets, check that the source IP matches the expected IP on the VLAN, then forward the packet according to the routing table. For inbound packets, lookup the destination IP and find the correct VLAN, then push the VLAN tag on the packet and forward it using the normal MAC lookup. For ARP packets, lookup the destination VLAN from the destination IP, change the VLAN tag and forward the packet. There is no reason a device should not be able to handle a large number of rules such as the above. The NoviSwitch will do it. However it appears that a lot of devices are quite limited in this regard. I could buy a router/switch for every few thousand users and split the work between them. Split the cost on many users, so the extra cost would probably not be prohibitive. This is the do the work at the edge solution, although I would be hosting the equipment in the same rack as the core router. But why fill a rack with equipment, to do simple dummy work, that should be manageable by a single device? Regards, Baldur
Re: The Making of a Router
On Sat, Dec 28, 2013 at 08:53:53AM -0600, Chris Adams wrote: > There is a significant value in "just plug it in and it works", and if > you don't figure your time investment (both up-front and on-going) into > the cost, you are greatly fooling yourself. What ISP-grade router are you using that is plug-and-play? So far, all the ones I've used have required some quite considerable configuration and testing to verify they do everything I need, and that's *after* I've spent a long time reading lies^Wfine marketing documentation trying to decide which combination of features best suits my needs, and then trying to guess what other features my business might need during the service lifetime of the device. - Matt -- I can only guess that the designer of the things had a major Toilet Duck habit and had managed to score a couple of industrial-sized bottles of the stuff the night before. -- Tanuki
Re: The Making of a Router
On Fri, Dec 27, 2013 at 08:47:25PM -0500, Jon Sands wrote: > On 12/27/2013 8:18 PM, Baldur Norddahl wrote: > >Brocade NetIron CER 2024F-4X goes for > >about $21k > > As one last aside, if you're paying 21k, you're paying a little more > than twice too much. Call Brocade and get yourself a real quote. s/a real quote/a referral to a reseller who takes a week to even return your call, let alone get a quote to you/ Been there, done that, got the T-shirt, about three months ago, and it was for quite a chunk more kit than $21k list. If they're so slack about getting around to taking my money, how sharp are they going to be when I need to cost them money (ask for support)? > I think peoples main point here is that any handful of thousand dollars > you think you will be saving by going PC based, is going to dissapear the > very first downtime you experience with zero support. So that ~$10k device comes with unlimited 24x7 30 minute response time support, which will get me in touch directly with the people who built and maintain the system I'm running, and with SLA penalties large enough to completely cover the additional costs I incur due to the vendor's failure to perform? That's quite a deal. - Matt -- A byte walks into a bar and orders a pint. Bartender asks him "What's wrong?" The byte says "Parity error." Bartender nods and says "Yeah, I thought you looked a bit off."
Re: The Making of a Router
> Pretty much what everyone else said. I'm a huge linux person, almost > everything I use is linux, run full Myth set up etc, but I wouldn't > use it for a high PPS situation like this. It's just asking for > suffering later, at the worst possible times. to paraphrase MO from many years ago (as i do not have a direct quote), do not use home appliances as production routers. there are niches where routeflow/vandervecken are usable. there are niches where quagga/bird/openbgp are useful. but not core/backbone of any non-trivial network. they might be usable for _slightly_ more than trivial if your business model values your time at zero. randy
Re: The Making of a Router
Pretty much what everyone else said. I'm a huge linux person, almost everything I use is linux, run full Myth set up etc, but I wouldn't use it for a high PPS situation like this. It's just asking for suffering later, at the worst possible times. -Blake On Sat, Dec 28, 2013 at 9:45 AM, Shawn Wilson wrote: > > > Chris Adams wrote: > >Once upon a time, Shawn Wilson said: > >> I was hoping someone could give technical insight into why this is > >good or not and not just "buy a box branded as a router because I said > >so or your business will fail". I'm all for hearing about the business > >theory of running an ISP (not my background or day job) but didn't > >think that's what the OP was asking about (and it didn't seem they were > >taking business suggestions very well anyway). > > > >There's been some technical insight here I would say. I'm a big Linux, > >Open Source, and Free Software advocate, and I'll use Linux-based > >systems for routing/firewalling small stuff, but for high speed/PPS, > >get > >a router with a hardware forwarding system (I like Juniper myself). > > > >You can build a decently-fast Linux (or *BSD) system, but you'll need > >to > >spend a good bit of time carefully choosing motherboards, cards, etc. > >to > >maximize packet handling, possibly buying multiple of each to find the > >best working combination. Make sure you buy a full set of spares once > >you find a working combination (because in the PC industry, six months > >is a lifetime). Then you have to build your OS install, tweaking the > >setup, network stack, etc. > > > >After that, you have to stay on top of updates and such (so plan for > >more reboots); while on a hardware-forwarding router you can mostly > >partition off the control plane, on a Linux/*BSD system, the base OS is > >the forwarding plane. Also, if something breaks, falls over under an > >attack, etc., you're generally going to be on your own to figure it > >out. > >Maybe you can Google the answer (and hope it isn't "that'll be fixed in > >kernel 3.. Not saying that doesn't happen with > >router vendors (quoting RFCs at router engineers is "fun"), but it is > >IMHO less often. > > > >The question becomes: what is your time worth? You could spend > >hundreds > >of hours going from the start to your satisfactory in-service router, > >and have a potentially higher upkeep cost. Can you hire somebody with > >all the same Linux/*BSD knowlege as yourself, so you are not on-call > >for > >your home-built router around the clock? > > > >I've used Linux on all my computers for almost 20 years, I develop on > >Linux, and contribute to a Linux distribution. However, when I want to > >record TV to watch later, I plug in a TiVo, not build a MythTV box. > >There is a significant value in "just plug it in and it works", and if > >you don't figure your time investment (both up-front and on-going) into > >the cost, you are greatly fooling yourself. > > I agree with all of this to some degree. IDK whether cost of ownership on > a hardware router or a desktop is more or less - I jus haven't done the > research. We use them at work and at home I have Cisco and Linksys gear > (plus Linux doing some things the router could like DHCP) - go figure. > > I agree that some network cards and boards work better than others (and am > partial to the Intel Pro cards - though I'm unsure if they're still the > best). I would also hesitate to route that much traffic with a PC. Though, > I have no technical reason for this bias. > > If you have hardware in production, you really should have a spare - > whether we're talking servers, HDDs, batteries, or routers. Ie, that > comment is not unique to servers. I also don't think warranty has any > bearing on this - I've seen servers stay down for over a day because (both > HP and Dell for their respective hardware) screwed up and the company > didn't budget for a spare board and I've seen a third of a network be taken > out because multiple switch ports just died. How much would a spare switch > have cost compared to 50 people not online? > > At any rate, I'm interested in this because I've worked in both > environments and haven't seen a large difference between the two approaches > (never worked at an ISP or high bandwidth web environment though). I do > like the PC router approach because it allows more versatility wrt dumping > packets (no need to dig out that 10mbit dumb hub and throttle the whole > network), I can run snort or do simple packet inspection with iptables > (some routers can do this but most can't or require a license). So I'm > sorta leaning to the PC router as being better - maybe not cheaper but > better. > >
Re: Mikrotik Cloud Core Router and BGP real life experiences?
MPLS has been one of Mikrotiks “selling points”. MPLS has been pretty stable for at least a year or more now. Their documentation has been kinda weak, but the implementation has been good. Justin -- Justin Wilson MTCNA CCNA MTCRE MTCWE - COMTRAIN Aol & Yahoo IM: j2sw http://www.mtin.net xISP News & Consulting http://www.zigwireless.com High Speed Internet Options http://www.thebrotherswisp.com The Brothers Wisp -Original Message- From: Seth Mattinen Date: Friday, December 27, 2013 at 1:19 PM To: Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? >On 12/27/13, 10:01, Justin Wilson wrote: >> The issues I see are because of routers versions. The Cloud core >>routers >> are a fairly new platform. As such, the software isn¹t as stable as it >> should be. The OS is up to version 6.7. There were some betas before >>6.0 >> was released. However, almost every version that has been released >> addresses issues with the cloud core. The cloud cores only run Version >>6. >> > > >Unless my knowledge is out of date, the one thing RouterOS has that >others in the same scope lack is a full MPLS stack that's not >experimental. > >~Seth >
Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
Out of all the network hardware I have worked on in operations these were by far some of the worst. I read lots of good things but like most things in life these just dont stack up against a Cisco or Juniper for stability and reliability. Most of the ISP's I have worked with were HSD but i also followed the progression path in the industry so i have time with Dial Up, ADSL/X/...,WISP's, Data Centers etc. and FTTH I generally only see these in WISP's and some DSL installs. Never anything with huge traffic load and full tables. Generally always driven by the cost factor alone without regard to much else imho. But that's just my experience. However maybe there are people that have managed to keep these up and handle all you have requested. just my 2c On Fri, Dec 27, 2013 at 10:00 AM, Dennis Burgess wrote: > We have many with full routing tables. Load balancing, works fine, I have > one site with 8 DSL lines doing balancing across them. We typically don't > use a GRE tunnel, but OpenVPN or IPSEC work great. > > > Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- > Second Edition" > Link Technologies, Inc -- Mikrotik & WISP Support > Services > Office: 314-735-0270 Website: http://www.linktechs.net - Skype: > linktechs > -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE > - 3G - 3.65 - TV Whitespace > > > -Original Message- > From: matt kelly [mailto:mjke...@gmail.com] > Sent: Friday, December 27, 2013 8:41 AM > To: Raymond Burkholder > Cc: NANOG list > Subject: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life > experiences? > > They can not handle a full routing table. The load balancing doesn't work. > They can not properly reassemble fragmented packets, and therefore drop > all but the first "piece". They can not reliably handle traffic loads over > maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel > connection. > > On Dec 27, 2013 9:07 AM, "Raymond Burkholder" wrote: > > > > > >My real world experience with these is that they suck. Plain and simple. > > >Don't waste your time. > > > > Would you mind elaborating what you were trying to accomplish and what > > failed? > > > > Thank you. > > > > Ray > > > > > > -- > > This message has been scanned for viruses and dangerous content by > > MailScanner, and is believed to be clean. > > > > > > > >
Re: The Making of a Router
Chris Adams wrote: >Once upon a time, Shawn Wilson said: >> I was hoping someone could give technical insight into why this is >good or not and not just "buy a box branded as a router because I said >so or your business will fail". I'm all for hearing about the business >theory of running an ISP (not my background or day job) but didn't >think that's what the OP was asking about (and it didn't seem they were >taking business suggestions very well anyway). > >There's been some technical insight here I would say. I'm a big Linux, >Open Source, and Free Software advocate, and I'll use Linux-based >systems for routing/firewalling small stuff, but for high speed/PPS, >get >a router with a hardware forwarding system (I like Juniper myself). > >You can build a decently-fast Linux (or *BSD) system, but you'll need >to >spend a good bit of time carefully choosing motherboards, cards, etc. >to >maximize packet handling, possibly buying multiple of each to find the >best working combination. Make sure you buy a full set of spares once >you find a working combination (because in the PC industry, six months >is a lifetime). Then you have to build your OS install, tweaking the >setup, network stack, etc. > >After that, you have to stay on top of updates and such (so plan for >more reboots); while on a hardware-forwarding router you can mostly >partition off the control plane, on a Linux/*BSD system, the base OS is >the forwarding plane. Also, if something breaks, falls over under an >attack, etc., you're generally going to be on your own to figure it >out. >Maybe you can Google the answer (and hope it isn't "that'll be fixed in >kernel 3.. Not saying that doesn't happen with >router vendors (quoting RFCs at router engineers is "fun"), but it is >IMHO less often. > >The question becomes: what is your time worth? You could spend >hundreds >of hours going from the start to your satisfactory in-service router, >and have a potentially higher upkeep cost. Can you hire somebody with >all the same Linux/*BSD knowlege as yourself, so you are not on-call >for >your home-built router around the clock? > >I've used Linux on all my computers for almost 20 years, I develop on >Linux, and contribute to a Linux distribution. However, when I want to >record TV to watch later, I plug in a TiVo, not build a MythTV box. >There is a significant value in "just plug it in and it works", and if >you don't figure your time investment (both up-front and on-going) into >the cost, you are greatly fooling yourself. I agree with all of this to some degree. IDK whether cost of ownership on a hardware router or a desktop is more or less - I jus haven't done the research. We use them at work and at home I have Cisco and Linksys gear (plus Linux doing some things the router could like DHCP) - go figure. I agree that some network cards and boards work better than others (and am partial to the Intel Pro cards - though I'm unsure if they're still the best). I would also hesitate to route that much traffic with a PC. Though, I have no technical reason for this bias. If you have hardware in production, you really should have a spare - whether we're talking servers, HDDs, batteries, or routers. Ie, that comment is not unique to servers. I also don't think warranty has any bearing on this - I've seen servers stay down for over a day because (both HP and Dell for their respective hardware) screwed up and the company didn't budget for a spare board and I've seen a third of a network be taken out because multiple switch ports just died. How much would a spare switch have cost compared to 50 people not online? At any rate, I'm interested in this because I've worked in both environments and haven't seen a large difference between the two approaches (never worked at an ISP or high bandwidth web environment though). I do like the PC router approach because it allows more versatility wrt dumping packets (no need to dig out that 10mbit dumb hub and throttle the whole network), I can run snort or do simple packet inspection with iptables (some routers can do this but most can't or require a license). So I'm sorta leaning to the PC router as being better - maybe not cheaper but better.
Re: The Making of a Router
Once upon a time, Shawn Wilson said: > I was hoping someone could give technical insight into why this is good or > not and not just "buy a box branded as a router because I said so or your > business will fail". I'm all for hearing about the business theory of running > an ISP (not my background or day job) but didn't think that's what the OP was > asking about (and it didn't seem they were taking business suggestions very > well anyway). There's been some technical insight here I would say. I'm a big Linux, Open Source, and Free Software advocate, and I'll use Linux-based systems for routing/firewalling small stuff, but for high speed/PPS, get a router with a hardware forwarding system (I like Juniper myself). You can build a decently-fast Linux (or *BSD) system, but you'll need to spend a good bit of time carefully choosing motherboards, cards, etc. to maximize packet handling, possibly buying multiple of each to find the best working combination. Make sure you buy a full set of spares once you find a working combination (because in the PC industry, six months is a lifetime). Then you have to build your OS install, tweaking the setup, network stack, etc. After that, you have to stay on top of updates and such (so plan for more reboots); while on a hardware-forwarding router you can mostly partition off the control plane, on a Linux/*BSD system, the base OS is the forwarding plane. Also, if something breaks, falls over under an attack, etc., you're generally going to be on your own to figure it out. Maybe you can Google the answer (and hope it isn't "that'll be fixed in kernel 3.. Not saying that doesn't happen with router vendors (quoting RFCs at router engineers is "fun"), but it is IMHO less often. The question becomes: what is your time worth? You could spend hundreds of hours going from the start to your satisfactory in-service router, and have a potentially higher upkeep cost. Can you hire somebody with all the same Linux/*BSD knowlege as yourself, so you are not on-call for your home-built router around the clock? I've used Linux on all my computers for almost 20 years, I develop on Linux, and contribute to a Linux distribution. However, when I want to record TV to watch later, I plug in a TiVo, not build a MythTV box. There is a significant value in "just plug it in and it works", and if you don't figure your time investment (both up-front and on-going) into the cost, you are greatly fooling yourself. -- Chris Adams
Re: The Making of a Router
In article you write: >It seems to be a pretty "hot button" issue, but I feel that modern hardware >is more than capable of pushing packets. The old wisdom of "only hardware >can do it efficiently" is starting to prove untrue. 10G might still be a >challenge (I haven't tested), but 1G is not even close to being an issue. > Depending on the target for your deployment, it might make sense to >whitebox a router or firewall instead of spending 20K on it. Especially if >you're working with any kind of scale. Yes well, but also remember that bandwidth is not everything. Packets per second is. And if you're going to provide internet connectivity to endusers, some of them /will/ get hit with DDOS attacks. With a hardware router you can survive that as long as the DDOS is not consuming all your bandwidth. A software router being bombarded with a few gigabits of 64 byte packets .. not so much. This is also the reason btw that you should look into shaping the outgoing bandwidth to each enduser, to prevent one of them being DDOSsed filling up the entire link he/she is on. Mike.
