RE: Mikrotik Cloud Core Router and BGP real life experiences?
As an update we put the first two into production on NYE Everything working as expected so far...
Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
"Dennis Burgess" writes: > Mikrotik really relies on its list of consultants and trainers, > these are all outside companies, yes such as mine, that provide the > higher class of "support" than MikroTik own e-mail. . While their > e-mail does have a lack of responsiveness, I was told the volume > that they do get form other parts of the world, not saying that's an > excuse, but it is what it is. This wasn't a support issue; it was bug reports. Things such as: * your CLI has an incomplete implementation of the Emacs key bindings (detailed list elided here on nanog@for brevity's sake but if you've ever used Mikrotik kit and are a seasoned CLI user on C and J platforms you know what I'm talking about); please consider fixing or adopting libcli, gnu readline, or somesuch in future releases. * your GRE implementation always has a protocol type of 0x0800 in the GRE header even when it is forwarding an IPv6 packet (packet dumps attached). * ssh sessions crash when ServerAliveInterval SSH application layer keepalives kick off. See http://www.openssh.org/faq.html section 2.12 or http://www.kehlet.cx/articles/129.html To replicate: ssh -o ServerAliveInterval=120 admin@myrouter (to their credit this was eventually fixed in 5.x - this behavior was observed in 5.0rc4) * /ping and /tool/traceroute fail for a DNS name for which there is no A record, only an record (although both commands will accept an IPv6 address as digits). This is still a problem today. * When trying to remove files, it seems that they are not removed by number, but rather by name, despite what the online help says. There was more stuff along those lines. "Thanks for the bug reports; I made sure to open tickets for them but we can't commit to when or if they'll get addressed due to competing priorities but they've absolutely been documented" would have been a fine reply; I completely understand the Real World considerations involved and that my priorities were not necessarily their priorities. Unfortunately the return email left me with the impression that nobody cared and that they were not equipped to handle issues brought to their attention by people with field experience, hence the unfavorable parallels to the "big guys". Note that this has not kept my from speccing their kit when the task calls for something that's surprisingly good considering how inexpensive it is! So maybe from a business perspective they were entirely correct to blow me off - at least where it comes to "revenue attributable to Rob Seastrom", the negative impact has been nil. -r
RE: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
Mikrotik really relies on its list of consultants and trainers, these are all outside companies, yes such as mine, that provide the higher class of "support" than MikroTik own e-mail. . While their e-mail does have a lack of responsiveness, I was told the volume that they do get form other parts of the world, not saying that's an excuse, but it is what it is. Many people in the WISP and smaller ISP markets rely on these consulting companies to not only help them with MikroTik but other hardware/software and business decisions, LTI (yes the company I work for) has more certified trainers and engineers for MikroTik than any other in North America, but there is a list from MikroTik that lists certified consultants available as well. Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net - Skype: linktechs -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE - 3G - 3.65 - TV Whitespace -Original Message- From: Rob Seastrom [mailto:r...@seastrom.com] Sent: Thursday, January 02, 2014 6:16 AM To: Justin Wilson Cc: NANOG list Subject: Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences? Justin Wilson writes: > The biggest problem with Mikrotik is you just can¹t call them up for > support on buggy code. In a critical network this can be a major problem. I've contacted them (via email) and the experience seems to be exactly the same as dealing with first level TAC at the big guys: the guy you contact doesn't care much about your problem once he realizes that it's a legitimate issue with their stuff and not simply a case of pilot error for which he can refer you to the documentation, and eventually you give up and develop a workaround, such as it is. -r
Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
Justin Wilson writes: > The biggest problem with Mikrotik is you just can¹t call them up for > support on buggy code. In a critical network this can be a major problem. I've contacted them (via email) and the experience seems to be exactly the same as dealing with first level TAC at the big guys: the guy you contact doesn't care much about your problem once he realizes that it's a legitimate issue with their stuff and not simply a case of pilot error for which he can refer you to the documentation, and eventually you give up and develop a workaround, such as it is. -r
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On Fri, Dec 27, 2013 at 6:47 AM, Martin Hotze wrote: > Hi, > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to good > to be true [1] having so much bang for the bucks. So virtually all smaller > ISPs would drop their CISCO gear for Mikrotik Routerboards. > The issue with RouterOS in general, not restricted to CCR's, is an annoying stuck route bug that seems to persist in 6.x versions. It causes packets to keep flowing to paths that are not working, routing protocols detect it but packets keep going there. Performance-wise, if your traffic is distributed among several ports you can benefit from having a core handling each port, but otherwise you should divide Mikrotik pps numbers to fit your scenario. Rubens
Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
The biggest problem with Mikrotik is you just can¹t call them up for support on buggy code. In a critical network this can be a major problem. Justin --- Justin Wilson MTIN Consulting Mikrotik UBNT Climbing Network Design http://www.mtin.net/ <http://www.mtin.net/blog> http://www.thebrotherswisp.com -Original Message- From: Dale Rumph Date: Friday, December 27, 2013 at 10:04 AM To: Dennis Burgess Cc: NANOG list Subject: Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences? >Out of all the network hardware I have worked on in operations these were >by far some of the worst. I read lots of good things but like most things >in life these just dont stack up against a Cisco or Juniper for stability >and reliability. Most of the ISP's I have worked with were HSD but i also >followed the progression path in the industry so i have time with Dial Up, >ADSL/X/...,WISP's, Data Centers etc. and FTTH > >I generally only see these in WISP's and some DSL installs. Never anything >with huge traffic load and full tables. Generally always driven by the >cost >factor alone without regard to much else imho. But that's just my >experience. However maybe there are people that have managed to keep these >up and handle all you have requested. > >just my 2c > > >On Fri, Dec 27, 2013 at 10:00 AM, Dennis Burgess >wrote: > >> We have many with full routing tables. Load balancing, works fine, I >>have >> one site with 8 DSL lines doing balancing across them. We typically >>don't >> use a GRE tunnel, but OpenVPN or IPSEC work great. >> >> >> Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- >> Second Edition" >> Link Technologies, Inc -- Mikrotik & WISP Support >> Services >> Office: 314-735-0270 Website: http://www.linktechs.net - Skype: >> linktechs >> -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE >> - 3G - 3.65 - TV Whitespace >> >> >> -Original Message----- >> From: matt kelly [mailto:mjke...@gmail.com] >> Sent: Friday, December 27, 2013 8:41 AM >> To: Raymond Burkholder >> Cc: NANOG list >> Subject: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life >> experiences? >> >> They can not handle a full routing table. The load balancing doesn't >>work. >> They can not properly reassemble fragmented packets, and therefore drop >> all but the first "piece". They can not reliably handle traffic loads >>over >> maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre >>tunnel >> connection. >> >> On Dec 27, 2013 9:07 AM, "Raymond Burkholder" >>wrote: >> >> > >> > >My real world experience with these is that they suck. Plain and >>simple. >> > >Don't waste your time. >> > >> > Would you mind elaborating what you were trying to accomplish and what >> > failed? >> > >> > Thank you. >> > >> > Ray >> > >> > >> > -- >> > This message has been scanned for viruses and dangerous content by >> > MailScanner, and is believed to be clean. >> > >> > >> > >> >> >
Re: Mikrotik Cloud Core Router and BGP real life experiences?
MPLS has been one of Mikrotiks “selling points”. MPLS has been pretty stable for at least a year or more now. Their documentation has been kinda weak, but the implementation has been good. Justin -- Justin Wilson MTCNA CCNA MTCRE MTCWE - COMTRAIN Aol & Yahoo IM: j2sw http://www.mtin.net xISP News & Consulting http://www.zigwireless.com High Speed Internet Options http://www.thebrotherswisp.com The Brothers Wisp -Original Message- From: Seth Mattinen Date: Friday, December 27, 2013 at 1:19 PM To: Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? >On 12/27/13, 10:01, Justin Wilson wrote: >> The issues I see are because of routers versions. The Cloud core >>routers >> are a fairly new platform. As such, the software isn¹t as stable as it >> should be. The OS is up to version 6.7. There were some betas before >>6.0 >> was released. However, almost every version that has been released >> addresses issues with the cloud core. The cloud cores only run Version >>6. >> > > >Unless my knowledge is out of date, the one thing RouterOS has that >others in the same scope lack is a full MPLS stack that's not >experimental. > >~Seth >
Re: [SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
Out of all the network hardware I have worked on in operations these were by far some of the worst. I read lots of good things but like most things in life these just dont stack up against a Cisco or Juniper for stability and reliability. Most of the ISP's I have worked with were HSD but i also followed the progression path in the industry so i have time with Dial Up, ADSL/X/...,WISP's, Data Centers etc. and FTTH I generally only see these in WISP's and some DSL installs. Never anything with huge traffic load and full tables. Generally always driven by the cost factor alone without regard to much else imho. But that's just my experience. However maybe there are people that have managed to keep these up and handle all you have requested. just my 2c On Fri, Dec 27, 2013 at 10:00 AM, Dennis Burgess wrote: > We have many with full routing tables. Load balancing, works fine, I have > one site with 8 DSL lines doing balancing across them. We typically don't > use a GRE tunnel, but OpenVPN or IPSEC work great. > > > Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- > Second Edition" > Link Technologies, Inc -- Mikrotik & WISP Support > Services > Office: 314-735-0270 Website: http://www.linktechs.net - Skype: > linktechs > -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE > - 3G - 3.65 - TV Whitespace > > > -Original Message- > From: matt kelly [mailto:mjke...@gmail.com] > Sent: Friday, December 27, 2013 8:41 AM > To: Raymond Burkholder > Cc: NANOG list > Subject: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life > experiences? > > They can not handle a full routing table. The load balancing doesn't work. > They can not properly reassemble fragmented packets, and therefore drop > all but the first "piece". They can not reliably handle traffic loads over > maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel > connection. > > On Dec 27, 2013 9:07 AM, "Raymond Burkholder" wrote: > > > > > >My real world experience with these is that they suck. Plain and simple. > > >Don't waste your time. > > > > Would you mind elaborating what you were trying to accomplish and what > > failed? > > > > Thank you. > > > > Ray > > > > > > -- > > This message has been scanned for viruses and dangerous content by > > MailScanner, and is believed to be clean. > > > > > > > >
Re: Mikrotik Cloud Core Router and BGP real life experiences?
Regards Alexander Alexander Neilson Neilson Productions Ltd alexan...@neilson.net.nz 021 329 681 > On 28/12/2013, at 5:06 am, Eduardo Schoedler wrote: > > PPPoE Server is single thread too. PPP package is getting a multicore upgrade in 6.8 or 6.9 release. May introduce bugs but they are working to Multi core all the processes properly. > > > 2013/12/27 Nick Olsen > >> Exactly what Faisal Said. The BGP process appears to be single threaded at >> the moment. So taking on full BGP tables can be a bit slow compared to a >> decent X86 box. But in terms of raw forwarding power they are pretty >> monstrous. >> >> We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K >> pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are >> in a configuration where they have little or no firewall/nat/queue rules. >> And in most cases are running MPLS. >> >> We've not had any issues with stability so far either (Knock on wood). >> >> Nick Olsen >> Network Operations >> (855) FLSPEED x106 >> >> >> From: "Faisal Imtiaz" >> Sent: Friday, December 27, 2013 10:33 AM >> To: "Geraint Jones" >> Cc: nanog@nanog.org, "Martin Hotze" >> Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? >> >> FYI... Mikrotik Cloud Core routers are nice, however one has to keep >> something in mind when deploying them... >> >> Only One Core (of the CPU) is dedicated to each port / process. >> So this is good so as to contain what happens on a single port from taxing >> the whole CPU.. >> But not so good when you need more cpu power than a single core for that >> port. >> >> Also, BGP process will only use one core. >> >> While these units make for great 'customer facing' edge routers, with >> plenty of power and the ability to keep issues contained... The X-86 based >> (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for >> running multiple full BGP tables peering. >> >> Regards & Good Luck. >> >> Faisal Imtiaz >> Snappy Internet & Telecom >> >> - Original Message - >>> From: "Geraint Jones" >>> To: "Martin Hotze" >>> Cc: nanog@nanog.org >>> Sent: Friday, December 27, 2013 4:02:45 AM >>> Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? >>> >>> I am going to be deploying 4 as edge routers in the next few weeks, each >> will >>> have 1 or 2 full tables plus partial IX tables. So I should have some >>> empirical info soon. >>> >>> They will be doing eBGP to upstreams and iBGP/OSPF internally. I went >> with >>> the 16gb RAM models. >>> >>> However these boxes are basically Linux running on top of tilera CPUs, >> in >>> terms of throughput as long as everything stays on the fastpath they have >> no >>> issues doing wire speed on all ports, however the moment you add a >> firewall >>> rule or the like they drop to 1.5gbps. >>> >>> >>> >>>> On 27/12/2013, at 9:47 pm, Martin Hotze wrote: >>>> >>>> Hi, >>>> >>>> looking at the specs of Mikrotik Cloud Core Routers it seems to be to >> good >>>> to be true [1] having so much bang for the bucks. So virtually all >> smaller >>>> ISPs would drop their CISCO gear for Mikrotik Routerboards. >>>> >>>> We are using a handful of Mikrotik boxes, but on a much lower network >> level >>>> (splitting networks; low end router behind ADSL modem, ...). We're >> happy >>>> with them. >>>> >>>> So I am asking for real life experience and not lab values with >> Mikrotik >>>> Cloud Core Routers and BGP. How good can they handle full tables and a >>>> bunch of peering sessions? How good does the box react when adding >> filters >>>> (during attacks)? Reloading the table? etc. etc. >>>> >>>> I am looking for _real_ _life_ values compared to a CISCO NPE-G2. >> Please >>>> tell me/us from your first hand experience. >>>> >>>> Thanks! >>>> >>>> greetings, Martin >>>> >>>> [1] If something sounds too good to be true, it probably is. > > > -- > Eduardo Schoedler smime.p7s Description: S/MIME cryptographic signature
Re: Mikrotik Cloud Core Router and BGP real life experiences?
How about SMP Affinity in CCR? System > Resources > IRQ. 2013/12/27 Andre Tomt > On 27. des. 2013 17:26, Jim Shankland wrote: > > > Routing table size was completely not an issue in our environment; we >> were looking at a number of concurrent flows in the high-5 to >> low-6-digit range, and since Linux uses a route cache, it was that >> number, rather than the number of full tables we carried, that was >> important. >> > > > FYI, Linux no longer has a routing cache, so any performance numbers with > the cache in place is void on modern kernels. It was deemed too fragile, > handled mixed traffic badly, and was way easy to DoS. It wasnt simply just > ripped out of course, the full lookups was made way faster and a bunch of > scalability issues got plugged in the process. > > All in all, in PPS, Linux should now handle mixed traffic much better, but > less diverse traffic patterns might be a little slower than before. > However, all in all, much more consistent and predictable. > > Not everything is peachy though, there are still some cases that sucked > last I checked. Running tons of tunnels beeing one. Multicast rx was > severely gimped for a while after the removal, but that got fixed. > > -- Eduardo Schoedler
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 27. des. 2013 17:26, Jim Shankland wrote: Routing table size was completely not an issue in our environment; we were looking at a number of concurrent flows in the high-5 to low-6-digit range, and since Linux uses a route cache, it was that number, rather than the number of full tables we carried, that was important. FYI, Linux no longer has a routing cache, so any performance numbers with the cache in place is void on modern kernels. It was deemed too fragile, handled mixed traffic badly, and was way easy to DoS. It wasnt simply just ripped out of course, the full lookups was made way faster and a bunch of scalability issues got plugged in the process. All in all, in PPS, Linux should now handle mixed traffic much better, but less diverse traffic patterns might be a little slower than before. However, all in all, much more consistent and predictable. Not everything is peachy though, there are still some cases that sucked last I checked. Running tons of tunnels beeing one. Multicast rx was severely gimped for a while after the removal, but that got fixed.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
Depends. This router isn't running BCP-38 as it's run at our borders. Looking at the specs. Firewall rules do take it out of fastpath. Which means it's going to take a decent performance hit on paper. I'm not sure if their auto method of enabling BCP-38 IE. the IP>Settings>RP Filter method would accomplish the same outcome, Without taking the router out of fastpath. I assume it works the same as using firewall rules. Just "Behind the scenes". That being said, Real world testing. Running the traffic levels I mentioned before. I put a single simple firewall rule on the router. Which effectively took it out of fastpath. And also enabled connection tracking. I saw no noticeable change in CPU load. Nick Olsen Network Operations (855) FLSPEED x106 From: "Hal Murray" Sent: Friday, December 27, 2013 2:38 PM To: nanog@nanog.org Cc: "Hal Murray" Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? nanog-requ...@nanog.org said: > We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K > pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are > in a configuration where they have little or no firewall/nat/queue rules. > And in most cases are running MPLS. How much CPU does it take to implement BCP-38? -- These are my opinions. I hate spam.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
nanog-requ...@nanog.org said: > We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K > pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are > in a configuration where they have little or no firewall/nat/queue rules. > And in most cases are running MPLS. How much CPU does it take to implement BCP-38? -- These are my opinions. I hate spam.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 12/27/13, 10:01, Justin Wilson wrote: The issues I see are because of routers versions. The Cloud core routers are a fairly new platform. As such, the software isn¹t as stable as it should be. The OS is up to version 6.7. There were some betas before 6.0 was released. However, almost every version that has been released addresses issues with the cloud core. The cloud cores only run Version 6. Unless my knowledge is out of date, the one thing RouterOS has that others in the same scope lack is a full MPLS stack that's not experimental. ~Seth
Re: Mikrotik Cloud Core Router and BGP real life experiences?
The issues I see are because of routers versions. The Cloud core routers are a fairly new platform. As such, the software isn¹t as stable as it should be. The OS is up to version 6.7. There were some betas before 6.0 was released. However, almost every version that has been released addresses issues with the cloud core. The cloud cores only run Version 6. We did se BGP issues early on accepting more than one full routing table. We saw other issues but they were fixed with subsequent OS software releases. Justin -- Justin Wilson MTCNA CCNA MTCRE MTCWE - COMTRAIN Aol & Yahoo IM: j2sw http://www.mtin.net/blog xISP News http://www.zigwireless.com High Speed Internet Options http://www.thebrotherswisp.com The Brothers Wisp -Original Message- From: Jim Shankland Date: Friday, December 27, 2013 at 11:26 AM To: Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? >On 12/27/13 6:40 AM, matt kelly wrote: >> They can not handle a full routing table. The load balancing doesn't >>work. >> They can not properly reassemble fragmented packets, and therefore drop >>all >> but the first "piece". They can not reliably handle traffic loads over >> maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre >>tunnel >> connection. >> >> >Can't say anything about MicroTik specifically, but I've used Linux as a >routing platform for many years, off and on, and took a reasonably close >look at performance about a year ago, in the previous job, using >relatively high-end, but pre-Sandy Bridge, generic hardware. We were >looking to support ca. 8 x 10 GbE ports with several full tables, and >the usual suspects wanted the usual 6-figure amounts for boxes that >could do that (the issue being the full routes -- 8 x 10 GbE with >minimal routing is a triviality these days). > >Routing table size was completely not an issue in our environment; we >were looking at a number of concurrent flows in the high-5 to >low-6-digit range, and since Linux uses a route cache, it was that >number, rather than the number of full tables we carried, that was >important. Doing store-and-forward packet processing, as opposed to >cut-through switching, took about 5 microseconds per packet, and >consumed about that much CPU time. The added latency was not an issue >for us; but at 5 us, that's 200Kpps per CPU. With 1500-byte packets, >that's about 2.4 Gb/s total throughput; but with 40-byte packets, it's >only 64 Mb/s (!). > >But that's per CPU. Our box had 24 CPUs (if you count a hyperthreaded >pair as 2), and this work is eminently parallelizable. So a theoretical >upper bound on throughput with this box would have been 4.8 Mpps -- 57.6 >Gb/s with 1500-byte packets, 1.5 Gb/s with 40-byte packets. > >The Linux network stack (plus RSS on the NICs) seemed to do quite a good >job of input-side parallelism - but we saw a lot of lock contention on >the output side. At that point, we abandoned the project, as it was >incidental to the organization's mission. I think that with a little >more work, we could have gotten within, say, a factor of 2 of the limits >above, which would have been good enough for us (though surely not for >everybody). Incrementally faster hardware would have incrementally >better performance. > >OpenFlow, which marries cheap, fast, and dumb ASICs with cheap, slower, >and infinitely flexible generic CPU and RAM, seemed, and still seems, >like the clearly right approach. At the time, it didn't seem ready for >prime time, either in the selection of OpenFlow-capable routers or in >the software stack. I imagine there's been some progress made since. >Whether the market will allow it to flourish is another question. > >Below a certain maximum throughput, routing with generic boxes is >actually pretty easy. Today, I'd say that maximum is roughly in the >low-single-gigabit range. Higher is possible, but gets progressively >harder to get right (and it's not a firm bound, anyway, as it depends on >traffic mix and other requirements). Whether it's worth doing really >depends on your goals and skill. Most people will probably prefer a >canned solution from a vendor. People who grow and eat their own food >surely eat better, and more cheaply, than those who buy at the >supermarket; but it's not for everybody. > >Jim Shankland > >
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 12/27/13 6:40 AM, matt kelly wrote: They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first "piece". They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. Can't say anything about MicroTik specifically, but I've used Linux as a routing platform for many years, off and on, and took a reasonably close look at performance about a year ago, in the previous job, using relatively high-end, but pre-Sandy Bridge, generic hardware. We were looking to support ca. 8 x 10 GbE ports with several full tables, and the usual suspects wanted the usual 6-figure amounts for boxes that could do that (the issue being the full routes -- 8 x 10 GbE with minimal routing is a triviality these days). Routing table size was completely not an issue in our environment; we were looking at a number of concurrent flows in the high-5 to low-6-digit range, and since Linux uses a route cache, it was that number, rather than the number of full tables we carried, that was important. Doing store-and-forward packet processing, as opposed to cut-through switching, took about 5 microseconds per packet, and consumed about that much CPU time. The added latency was not an issue for us; but at 5 us, that's 200Kpps per CPU. With 1500-byte packets, that's about 2.4 Gb/s total throughput; but with 40-byte packets, it's only 64 Mb/s (!). But that's per CPU. Our box had 24 CPUs (if you count a hyperthreaded pair as 2), and this work is eminently parallelizable. So a theoretical upper bound on throughput with this box would have been 4.8 Mpps -- 57.6 Gb/s with 1500-byte packets, 1.5 Gb/s with 40-byte packets. The Linux network stack (plus RSS on the NICs) seemed to do quite a good job of input-side parallelism - but we saw a lot of lock contention on the output side. At that point, we abandoned the project, as it was incidental to the organization's mission. I think that with a little more work, we could have gotten within, say, a factor of 2 of the limits above, which would have been good enough for us (though surely not for everybody). Incrementally faster hardware would have incrementally better performance. OpenFlow, which marries cheap, fast, and dumb ASICs with cheap, slower, and infinitely flexible generic CPU and RAM, seemed, and still seems, like the clearly right approach. At the time, it didn't seem ready for prime time, either in the selection of OpenFlow-capable routers or in the software stack. I imagine there's been some progress made since. Whether the market will allow it to flourish is another question. Below a certain maximum throughput, routing with generic boxes is actually pretty easy. Today, I'd say that maximum is roughly in the low-single-gigabit range. Higher is possible, but gets progressively harder to get right (and it's not a firm bound, anyway, as it depends on traffic mix and other requirements). Whether it's worth doing really depends on your goals and skill. Most people will probably prefer a canned solution from a vendor. People who grow and eat their own food surely eat better, and more cheaply, than those who buy at the supermarket; but it's not for everybody. Jim Shankland
Re: Mikrotik Cloud Core Router and BGP real life experiences?
PPPoE Server is single thread too. 2013/12/27 Nick Olsen > Exactly what Faisal Said. The BGP process appears to be single threaded at > the moment. So taking on full BGP tables can be a bit slow compared to a > decent X86 box. But in terms of raw forwarding power they are pretty > monstrous. > > We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K > pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are > in a configuration where they have little or no firewall/nat/queue rules. > And in most cases are running MPLS. > > We've not had any issues with stability so far either (Knock on wood). > > Nick Olsen > Network Operations > (855) FLSPEED x106 > > > From: "Faisal Imtiaz" > Sent: Friday, December 27, 2013 10:33 AM > To: "Geraint Jones" > Cc: nanog@nanog.org, "Martin Hotze" > Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? > > FYI... Mikrotik Cloud Core routers are nice, however one has to keep > something in mind when deploying them... > > Only One Core (of the CPU) is dedicated to each port / process. > So this is good so as to contain what happens on a single port from taxing > the whole CPU.. > But not so good when you need more cpu power than a single core for that > port. > > Also, BGP process will only use one core. > > While these units make for great 'customer facing' edge routers, with > plenty of power and the ability to keep issues contained... The X-86 based > (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for > running multiple full BGP tables peering. > > Regards & Good Luck. > > Faisal Imtiaz > Snappy Internet & Telecom > > ----- Original Message - > > From: "Geraint Jones" > > To: "Martin Hotze" > > Cc: nanog@nanog.org > > Sent: Friday, December 27, 2013 4:02:45 AM > > Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? > > > > I am going to be deploying 4 as edge routers in the next few weeks, each > will > > have 1 or 2 full tables plus partial IX tables. So I should have some > > empirical info soon. > > > > They will be doing eBGP to upstreams and iBGP/OSPF internally. I went > with > > the 16gb RAM models. > > > > However these boxes are basically Linux running on top of tilera CPUs, > in > > terms of throughput as long as everything stays on the fastpath they have > no > > issues doing wire speed on all ports, however the moment you add a > firewall > > rule or the like they drop to 1.5gbps. > > > > > > > > > On 27/12/2013, at 9:47 pm, Martin Hotze wrote: > > > > > > Hi, > > > > > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to > good > > > to be true [1] having so much bang for the bucks. So virtually all > smaller > > > ISPs would drop their CISCO gear for Mikrotik Routerboards. > > > > > > We are using a handful of Mikrotik boxes, but on a much lower network > level > > > (splitting networks; low end router behind ADSL modem, ...). We're > happy > > > with them. > > > > > > So I am asking for real life experience and not lab values with > Mikrotik > > > Cloud Core Routers and BGP. How good can they handle full tables and a > > > bunch of peering sessions? How good does the box react when adding > filters > > > (during attacks)? Reloading the table? etc. etc. > > > > > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. > Please > > > tell me/us from your first hand experience. > > > > > > Thanks! > > > > > > greetings, Martin > > > > > > [1] If something sounds too good to be true, it probably is. > > > > > > > > > > > > > > > > -- Eduardo Schoedler
Re: Mikrotik Cloud Core Router and BGP real life experiences?
Exactly what Faisal Said. The BGP process appears to be single threaded at the moment. So taking on full BGP tables can be a bit slow compared to a decent X86 box. But in terms of raw forwarding power they are pretty monstrous. We replaced a few Maxxwave 6 port Atom's with the CCR. ~400Mb/s and ~40K pps aggregate across all ports. CPU load went from ~25% to ~0-2%. These are in a configuration where they have little or no firewall/nat/queue rules. And in most cases are running MPLS. We've not had any issues with stability so far either (Knock on wood). Nick Olsen Network Operations (855) FLSPEED x106 From: "Faisal Imtiaz" Sent: Friday, December 27, 2013 10:33 AM To: "Geraint Jones" Cc: nanog@nanog.org, "Martin Hotze" Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? FYI... Mikrotik Cloud Core routers are nice, however one has to keep something in mind when deploying them... Only One Core (of the CPU) is dedicated to each port / process. So this is good so as to contain what happens on a single port from taxing the whole CPU.. But not so good when you need more cpu power than a single core for that port. Also, BGP process will only use one core. While these units make for great 'customer facing' edge routers, with plenty of power and the ability to keep issues contained... The X-86 based (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for running multiple full BGP tables peering. Regards & Good Luck. Faisal Imtiaz Snappy Internet & Telecom - Original Message - > From: "Geraint Jones" > To: "Martin Hotze" > Cc: nanog@nanog.org > Sent: Friday, December 27, 2013 4:02:45 AM > Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? > > I am going to be deploying 4 as edge routers in the next few weeks, each will > have 1 or 2 full tables plus partial IX tables. So I should have some > empirical info soon. > > They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with > the 16gb RAM models. > > However these boxes are basically Linux running on top of tilera CPUs, in > terms of throughput as long as everything stays on the fastpath they have no > issues doing wire speed on all ports, however the moment you add a firewall > rule or the like they drop to 1.5gbps. > > > > > On 27/12/2013, at 9:47 pm, Martin Hotze wrote: > > > > Hi, > > > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to good > > to be true [1] having so much bang for the bucks. So virtually all smaller > > ISPs would drop their CISCO gear for Mikrotik Routerboards. > > > > We are using a handful of Mikrotik boxes, but on a much lower network level > > (splitting networks; low end router behind ADSL modem, ...). We're happy > > with them. > > > > So I am asking for real life experience and not lab values with Mikrotik > > Cloud Core Routers and BGP. How good can they handle full tables and a > > bunch of peering sessions? How good does the box react when adding filters > > (during attacks)? Reloading the table? etc. etc. > > > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please > > tell me/us from your first hand experience. > > > > Thanks! > > > > greetings, Martin > > > > [1] If something sounds too good to be true, it probably is. > > > > > > > >
Re: Mikrotik Cloud Core Router and BGP real life experiences?
FYI... Mikrotik Cloud Core routers are nice, however one has to keep something in mind when deploying them... Only One Core (of the CPU) is dedicated to each port / process. So this is good so as to contain what happens on a single port from taxing the whole CPU.. But not so good when you need more cpu power than a single core for that port. Also, BGP process will only use one core. While these units make for great 'customer facing' edge routers, with plenty of power and the ability to keep issues contained... The X-86 based (Core2Duo/i5/i7) Mikrotik are more suitable (Processing power wise) for running multiple full BGP tables peering. Regards & Good Luck. Faisal Imtiaz Snappy Internet & Telecom - Original Message - > From: "Geraint Jones" > To: "Martin Hotze" > Cc: nanog@nanog.org > Sent: Friday, December 27, 2013 4:02:45 AM > Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? > > I am going to be deploying 4 as edge routers in the next few weeks, each will > have 1 or 2 full tables plus partial IX tables. So I should have some > empirical info soon. > > They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with > the 16gb RAM models. > > However these boxes are basically Linux running on top of tilera CPUs, in > terms of throughput as long as everything stays on the fastpath they have no > issues doing wire speed on all ports, however the moment you add a firewall > rule or the like they drop to 1.5gbps. > > > > > On 27/12/2013, at 9:47 pm, Martin Hotze wrote: > > > > Hi, > > > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to good > > to be true [1] having so much bang for the bucks. So virtually all smaller > > ISPs would drop their CISCO gear for Mikrotik Routerboards. > > > > We are using a handful of Mikrotik boxes, but on a much lower network level > > (splitting networks; low end router behind ADSL modem, ...). We're happy > > with them. > > > > So I am asking for real life experience and not lab values with Mikrotik > > Cloud Core Routers and BGP. How good can they handle full tables and a > > bunch of peering sessions? How good does the box react when adding filters > > (during attacks)? Reloading the table? etc. etc. > > > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please > > tell me/us from your first hand experience. > > > > Thanks! > > > > greetings, Martin > > > > [1] If something sounds too good to be true, it probably is. > > > > > > > >
[SPAM]RE: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences?
We have many with full routing tables. Load balancing, works fine, I have one site with 8 DSL lines doing balancing across them. We typically don't use a GRE tunnel, but OpenVPN or IPSEC work great. Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net - Skype: linktechs -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE - 3G - 3.65 - TV Whitespace -Original Message- From: matt kelly [mailto:mjke...@gmail.com] Sent: Friday, December 27, 2013 8:41 AM To: Raymond Burkholder Cc: NANOG list Subject: [SPAM]RE: Mikrotik Cloud Core Router and BGP real life experiences? They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first "piece". They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. On Dec 27, 2013 9:07 AM, "Raymond Burkholder" wrote: > > >My real world experience with these is that they suck. Plain and simple. > >Don't waste your time. > > Would you mind elaborating what you were trying to accomplish and what > failed? > > Thank you. > > Ray > > > -- > This message has been scanned for viruses and dangerous content by > MailScanner, and is believed to be clean. > > >
RE: Mikrotik Cloud Core Router and BGP real life experiences?
I too am curious... We've used them for a few months as edge devices and most (if not all) *knock on wood* of the issues we've had have been fixed by RouterOS updates, configuration changes (lots of chefs in the kitchen), or were circuit/carrier related. While I would never compare them apples-to-apples to Cisco, Juniper, etc devices... they have, in our experience, proven to be good inexpensive routers with a few quirks here and there. From: Raymond Burkholder [r...@oneunified.net] Sent: Friday, December 27, 2013 9:05 AM To: 'NANOG list' Subject: RE: Mikrotik Cloud Core Router and BGP real life experiences? >My real world experience with these is that they suck. Plain and simple. >Don't waste your time. Would you mind elaborating what you were trying to accomplish and what failed? Thank you. Ray -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
RE: Mikrotik Cloud Core Router and BGP real life experiences?
They can not handle a full routing table. The load balancing doesn't work. They can not properly reassemble fragmented packets, and therefore drop all but the first "piece". They can not reliably handle traffic loads over maybe 200 Mbps, we needed 4-6 Gbps capacity. They can not hold a gre tunnel connection. On Dec 27, 2013 9:07 AM, "Raymond Burkholder" wrote: > > >My real world experience with these is that they suck. Plain and simple. > >Don't waste your time. > > Would you mind elaborating what you were trying to accomplish and what > failed? > > Thank you. > > Ray > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > >
[SPAM]RE: [SPAM]Re: Mikrotik Cloud Core Router and BGP real life experiences?
Guess I should chime in here. As far as the CCR, I know several customers running in excess of 1 gig of traffic though them, one has 16 BGP sessions, several of those are full tables, and the rest are on an peering exchange. There are other units, like the ones we supply, that does more than 20 gig in real word usages. They are very capable devices, but depending on how many features you enable, of course that will affect their overall abilities. This would be real word, and yes, I work with 1000's of ISPs across North America, many between 100-10gig of traffic, cable companies, DSL providers, and WISPs, and many of these ONLY use MikroTik. As another person said, grab two and configure so that you split your load up, we have done that in areas where redundancy is important. Seeing the Dual 10GigE model with 8 GigE ports costs $1,249 or so, hard to beat them in price, and add two or more to get your redundancy. Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net - Skype: linktechs -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE - 3G - 3.65 - TV Whitespace -Original Message- From: Eduardo Schoedler [mailto:lis...@esds.com.br] Sent: Friday, December 27, 2013 8:10 AM To: NANOG list Subject: [SPAM]Re: Mikrotik Cloud Core Router and BGP real life experiences? People who tested say they don't forward more than 500Mbps per port. 2013/12/27 matt kelly > My real world experience with these is that they suck. Plain and simple. > Don't waste your time. > On Dec 27, 2013 3:49 AM, "Martin Hotze" wrote: > > > Hi, > > > > looking at the specs of Mikrotik Cloud Core Routers it seems to be > > to > good > > to be true [1] having so much bang for the bucks. So virtually all > smaller > > ISPs would drop their CISCO gear for Mikrotik Routerboards. > > > > We are using a handful of Mikrotik boxes, but on a much lower > > network level (splitting networks; low end router behind ADSL modem, > > ...). We're happy with them. > > > > So I am asking for real life experience and not lab values with > > Mikrotik Cloud Core Routers and BGP. How good can they handle full > > tables and a bunch of peering sessions? How good does the box react > > when adding > filters > > (during attacks)? Reloading the table? etc. etc. > > > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. > > Please tell me/us from your first hand experience. > > > > Thanks! > > > > greetings, Martin > > > > [1] If something sounds too good to be true, it probably is. > > > > > > > > > -- Eduardo Schoedler
Re: Mikrotik Cloud Core Router and BGP real life experiences?
People who tested say they don't forward more than 500Mbps per port. 2013/12/27 matt kelly > My real world experience with these is that they suck. Plain and simple. > Don't waste your time. > On Dec 27, 2013 3:49 AM, "Martin Hotze" wrote: > > > Hi, > > > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to > good > > to be true [1] having so much bang for the bucks. So virtually all > smaller > > ISPs would drop their CISCO gear for Mikrotik Routerboards. > > > > We are using a handful of Mikrotik boxes, but on a much lower network > > level (splitting networks; low end router behind ADSL modem, ...). We're > > happy with them. > > > > So I am asking for real life experience and not lab values with Mikrotik > > Cloud Core Routers and BGP. How good can they handle full tables and a > > bunch of peering sessions? How good does the box react when adding > filters > > (during attacks)? Reloading the table? etc. etc. > > > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please > > tell me/us from your first hand experience. > > > > Thanks! > > > > greetings, Martin > > > > [1] If something sounds too good to be true, it probably is. > > > > > > > > > -- Eduardo Schoedler
RE: Mikrotik Cloud Core Router and BGP real life experiences?
>My real world experience with these is that they suck. Plain and simple. >Don't waste your time. Would you mind elaborating what you were trying to accomplish and what failed? Thank you. Ray -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Mikrotik Cloud Core Router and BGP real life experiences?
My real world experience with these is that they suck. Plain and simple. Don't waste your time. On Dec 27, 2013 3:49 AM, "Martin Hotze" wrote: > Hi, > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to good > to be true [1] having so much bang for the bucks. So virtually all smaller > ISPs would drop their CISCO gear for Mikrotik Routerboards. > > We are using a handful of Mikrotik boxes, but on a much lower network > level (splitting networks; low end router behind ADSL modem, ...). We're > happy with them. > > So I am asking for real life experience and not lab values with Mikrotik > Cloud Core Routers and BGP. How good can they handle full tables and a > bunch of peering sessions? How good does the box react when adding filters > (during attacks)? Reloading the table? etc. etc. > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please > tell me/us from your first hand experience. > > Thanks! > > greetings, Martin > > [1] If something sounds too good to be true, it probably is. > > > >
Re: Mikrotik Cloud Core Router and BGP real life experiences?
On 12/27/2013 05:59 AM, Martin Hotze wrote: On 27/12/2013, at 10:13 pm, Martin Hotze wrote: Thanks, estimated traffic levels are at about half a gig, but at least 50 megs of UDP (VoIP) in both directions. one thing is that I haven't found a solution for redundant power supply. Buy 2 :) on 3am I only want to read the notification and know what to do first in the morning. And not jump out and bring the spare into production. #m You set them both up configure the spare for fail-over.
RE: Mikrotik Cloud Core Router and BGP real life experiences?
> > On 27/12/2013, at 10:13 pm, Martin Hotze wrote: > > > > Thanks, > > > > estimated traffic levels are at about half a gig, but at least 50 megs > of UDP (VoIP) in both directions. > > > > one thing is that I haven't found a solution for redundant power supply. > > > Buy 2 :) on 3am I only want to read the notification and know what to do first in the morning. And not jump out and bring the spare into production. #m
Re: Mikrotik Cloud Core Router and BGP real life experiences?
> On 27/12/2013, at 10:13 pm, Martin Hotze wrote: > > Thanks, > > estimated traffic levels are at about half a gig, but at least 50 megs of UDP > (VoIP) in both directions. > > one thing is that I haven't found a solution for redundant power supply. > Buy 2 :) > #m > >> -Original Message- >> From: Geraint Jones [mailto:gera...@koding.com] >> Sent: Friday, December 27, 2013 10:03 AM >> To: Martin Hotze >> Cc: nanog@nanog.org >> Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? >> >> I am going to be deploying 4 as edge routers in the next few weeks, each >> will have 1 or 2 full tables plus partial IX tables. So I should have some >> empirical info soon. >> >> They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with >> the 16gb RAM models. >> >> However these boxes are basically Linux running on top of tilera CPUs, in >> terms of throughput as long as everything stays on the fastpath they have >> no issues doing wire speed on all ports, however the moment you add a >> firewall rule or the like they drop to 1.5gbps. >> >> >> >>> On 27/12/2013, at 9:47 pm, Martin Hotze wrote: >> (...)
RE: Mikrotik Cloud Core Router and BGP real life experiences?
Thanks, estimated traffic levels are at about half a gig, but at least 50 megs of UDP (VoIP) in both directions. one thing is that I haven't found a solution for redundant power supply. #m > -Original Message- > From: Geraint Jones [mailto:gera...@koding.com] > Sent: Friday, December 27, 2013 10:03 AM > To: Martin Hotze > Cc: nanog@nanog.org > Subject: Re: Mikrotik Cloud Core Router and BGP real life experiences? > > I am going to be deploying 4 as edge routers in the next few weeks, each > will have 1 or 2 full tables plus partial IX tables. So I should have some > empirical info soon. > > They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with > the 16gb RAM models. > > However these boxes are basically Linux running on top of tilera CPUs, in > terms of throughput as long as everything stays on the fastpath they have > no issues doing wire speed on all ports, however the moment you add a > firewall rule or the like they drop to 1.5gbps. > > > > > On 27/12/2013, at 9:47 pm, Martin Hotze wrote: > > > (...)
Re: Mikrotik Cloud Core Router and BGP real life experiences?
I am going to be deploying 4 as edge routers in the next few weeks, each will have 1 or 2 full tables plus partial IX tables. So I should have some empirical info soon. They will be doing eBGP to upstreams and iBGP/OSPF internally. I went with the 16gb RAM models. However these boxes are basically Linux running on top of tilera CPUs, in terms of throughput as long as everything stays on the fastpath they have no issues doing wire speed on all ports, however the moment you add a firewall rule or the like they drop to 1.5gbps. > On 27/12/2013, at 9:47 pm, Martin Hotze wrote: > > Hi, > > looking at the specs of Mikrotik Cloud Core Routers it seems to be to good to > be true [1] having so much bang for the bucks. So virtually all smaller ISPs > would drop their CISCO gear for Mikrotik Routerboards. > > We are using a handful of Mikrotik boxes, but on a much lower network level > (splitting networks; low end router behind ADSL modem, ...). We're happy with > them. > > So I am asking for real life experience and not lab values with Mikrotik > Cloud Core Routers and BGP. How good can they handle full tables and a bunch > of peering sessions? How good does the box react when adding filters (during > attacks)? Reloading the table? etc. etc. > > I am looking for _real_ _life_ values compared to a CISCO NPE-G2. Please tell > me/us from your first hand experience. > > Thanks! > > greetings, Martin > > [1] If something sounds too good to be true, it probably is. > > >
