I should also add the every CLI option is configurable via GUI on the ConfigTree tab.
Conntrack values can be tweaked in System - > Connntrack. On Fri, Jun 10, 2016 at 1:28 PM, Andrew Duey < andrew.d...@widerangebroadband.net> wrote: > Scott, > To answer your questions, I do not have any junipers or real Cisco > units in the network. I've worked with a few smaller Cisco units but > nothing of even the same caliber. I only have a handful of the EdgeRouter > Pro's at each site I support. For the ISP network we have a pair of > EdgeRouter Pro's at the core and then have Mikrotik at all the tower > sites. We had been using Vyatta for years before Brocade bought them and > burnt all bridges with me (what a terrible, worthless, corporate greed > driven company). > > --Andrew > > > On Friday, June 10, 2016, Scott Lambert <lamb...@lambertfam.org> wrote: > >> Thank you for the reply. I would not keep attempting to make the WebUI >> work, except the guy paying me to set these up specifically does not >> want to type more than he has to. He's not much of a typist. >> >> If i could find an equivalent of RouterOS's "/interface monitor-traffic >> ether1" or cisco's "show interfaces GigabitEthernet 0/3", I would never >> load EdgeOS's heavy WebUI. >> >> I just compared the connection tracking timeouts between EdgeOS and >> RouterOS. >> >> It looks to me like RouterOS is at least three times as aggressive >> about getting rid of connections which were likely to be malicious >> traffic. It also looks to me as though RouterOS autoscales the size of >> the connection tracking table when it gets "too full"; whatever "too >> full" means. >> >> The minimum size I've seen is 88088 entries, the maximum on my network >> is somewhere over 472,000 entries on a box which has lots of firewall >> rules, does NAT, and passes 800Mbps during peak hours for more than 1500 >> users. That box is using less than 1GB of RAM. Maybe I'll just set all >> of the EdgeRouters to use similar settings to that CCR1036. >> >> Just out of curiosity, how many EdgeRouters do you have deployed? >> >> Do you use anything between the Juniper and the EdgeRouter Pro? >> >> I'm at around 100 RouterOS routers in the ISP network. Several more >> at client businesses. In ISP networks, I've only used about 10 >> ImageStreams and a few dozen Ciscos. No Junipers, yet. >> >> The Ciscos are hands down the reliability champs, followed by >> ImageStream, then RouterOS, then the EdgeRouters, in my experience so >> far. >> >> Like you said, it's hard to justify the big brands when MikroTik and >> Ubiquiti have such low cost options in the CCR1009s and the EdgeRouter >> Pros. >> >> I better get back to looking up how to setup GRE tunnels across the >> Internet and run OSPF across them as a backup path for when the network >> gets partitioned internally on the EdgeRouters. >> >> On Thu, Jun 09, 2016 at 11:05:03PM -0500, Andrew Duey wrote: >> > Scott, >> > >> > We have been running the EdgeRouter Pro's for at least a year now and >> ran >> > Vyatta for a couple years before that. After reading your frustrations >> > here's a couple observations: >> > >> > 1) We do NO NAT in our Edgerouters now, but when we did in Vyatta we had >> > the same conntrack issues. We found you could increase the table size >> > dramatically and that fixed our problems until we could un-NAT >> everything. >> > I would recommend not doing NAT on those units at all but if you have to >> > you can increase the table sizes to help. >> > >> > 2) Ours have been crazy stable (given the very low cost of the units). >> > >> > 3) The web gui is only useful for viewing real-time network >> information. I >> > do like that you can see real-time traffic counts, see how routing >> changes >> > take effect, and see an outage in a heartbeat but the web guy is for >> > information only. I would STRONGLY recommend that in an ISP environment >> > that ALL configuration be done at the command line. I found it very >> > friendly and easy to use, and there used to be TONS of peer Vyatta >> support >> > out there (Brocade killed a lot of it, but there's still a lot left in >> > other places). >> > >> > 4) While UBNT's direct support is crap on a good day, I've found that >> their >> > forums have 90% of the answers I'm looking for and the rest can be >> gotten >> > with my googlefu and searching for Vyatta examples. >> > >> > While the $400 EdgeRouter Pro is no Juniper MX160 I've found that they >> are >> > amazing little boxes given their price point. We make a point to always >> > keep at least one spare on hand and run them in pairs anyway but so far >> > haven't needed to worry about them too much (knock on wood). >> > >> > Best of luck with them >> > --Andrew Duey >> > >> > On Thursday, June 9, 2016, Scott Lambert <lamb...@lambertfam.org> >> wrote: >> > >> > > Warning, this message has not been well editted. I don't have time to >> > > go back and get it to try to get the tone to match what is intended. >> I >> > > have limitted time onsite during which to try to fix issues. However >> > > "rant"y this sounds, it should sound like frustration rather than >> anger. >> > > >> > > I am helping an ISP owner who is trying to transition from one big >> > > bridge to a mostly routed network. He had ImageStream routers at >> three >> > > points, two points were BGP upstreams where there is not really enough >> > > bandwidth between the sites to fail over gracefully. We are working >> on >> > > that too. >> > > >> > > Last summer, he started running into devices falling over because the >> > > bridge table was full, so we tried putting an EdgeRouter at one site >> > > toward the long end of the network to get about 400 devices off the >> main >> > > bridge as a quick bandaid. It fell over about the time he got an hour >> > > away from the site. We found conntrack issues in the log. We didn't >> > > have any firewall rules. We rebooted and it ran fine, for an hour or >> > > so. Then more conntrack issues. We quickly configured a spare >> 100Mbps >> > > ImageStream and had no more issues. 100Mbps isn't really a long term >> > > option but was okay at the time. >> > > >> > > In October, I came on-site and we built and installed another >> EdgeRouter >> > > PoE because we were half afraid of the original hardware. This time >> > > v1.7.0 was available and we got it running without blowing up. At >> least >> > > for the week I was on-site. We also installed six other EdgeRouters >> > > to replace the two BGP speaking ImageStreams and to segment four other >> > > hub points in the network. Yea! no more bridge loops without turning >> > > toughswitch ports on and off. OSPF fails over properly. Everything >> > > looked okay while I was onsite. >> > > >> > > Since then, there have been a few times when no traffic would pass >> > > through one of those hub sites and he would have to powercycle >> > > everything. >> > > >> > > Then I happened to be on-site for the AirOS MF worm outbreak. Two of >> > > the hub sites repeatedly stopped passing traffic while we were trying >> > > to clean up. Being onsite I was able to be patient and persistent and >> > > eventually get into the router at the site which was offline to view >> the >> > > log. The logs said the connection tracking table was full. Sometimes >> > > my connection to the router would continue to work long enough to >> issue >> > > a reboot command after seeing the logs. Sometimes I would have to be >> > > patient and persistent to get back into the router to issue the >> reboot. >> > > I tried to blame it on the AirOS MF worm traffic. >> > > >> > > We replaced his last ImageStream router with an EdgeRouter Pro the day >> > > before I left after cleaning up the AirOS MF worm. We put 1.8.0 on >> > > it and didn't expect any issues. Two hours after it started passing >> > > traffic, it overflowed the conntrack tables. We heard from customers >> at >> > > 7 am that it had been down since 1:30 am. He moved the cables back to >> > > the still powered Imagestream and all was well for the next month. I >> > > came back and looked at the log on the still running EdgeRouter Pro >> and >> > > it has nothing but "nf_conntrack: table full, dropping packet" errors >> > > from 1:17 to 07:13. >> > > >> > > Shouldn't it have recovered at some point during the six hours? The >> > > rest of the network was fine during this time. Connection tracking >> had >> > > no issues on the BGP router which handles this traffic + the other 4 >> > > times this number of clients traffic, there were no issues with the >> two >> > > hub routers which pass traffic in the ring from the BGP router to this >> > > router. >> > > >> > > #Powered up, cables connected: >> > > Jan 1 00:24:15 RouterName1 OSPF[906]: OSPF-6: Message-digest-key >> key-id >> > > 1 enter Passive mode >> > > Jan 1 00:25:29 RouterName1 OSPF[906]: OSPF-6: Message-digest-key >> key-id >> > > 1 enter Passive mode >> > > May 19 22:56:41 RouterName1 OSPF[906]: OSPF-4: Detected router with >> > > duplicate router ID10.11.100.2 in area 0.0.0.0 >> > > >> > > # I am disappointed that there were no link up messages as we moved >> the >> > > # cables to this device. The last OSPF speaking cable was moved after >> > > # the above log message. The EdgeRouter uses the same router ID as >> the >> > > # Imagestream. So the above message was expected. >> > > >> > > May 20 00:17:01 RouterName1 rsyslogd: set SCM_CREDENTIALS failed on >> > > '/dev/log': Protocol not avai >> > > lable >> > > May 20 01:17:26 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 01:17:31 RouterName1 kernel: last message repeated 9 times >> > > May 20 01:17:31 RouterName1 kernel: net_ratelimit: 87 callbacks >> suppressed >> > > May 20 01:17:31 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 01:17:36 RouterName1 kernel: last message repeated 9 times >> > > May 20 01:17:36 RouterName1 kernel: net_ratelimit: 73 callbacks >> suppressed >> > > May 20 01:17:36 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 01:17:41 RouterName1 kernel: last message repeated 9 times >> > > May 20 01:17:41 RouterName1 kernel: net_ratelimit: 118 callbacks >> suppressed >> > > May 20 01:17:41 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 01:17:46 RouterName1 kernel: last message repeated 9 times >> > > May 20 01:17:46 RouterName1 kernel: net_ratelimit: 262 callbacks >> suppressed >> > > May 20 01:17:46 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 01:17:51 RouterName1 kernel: last message repeated 9 times >> > > >> > > #Last ethernet cord being pulled: >> > > May 20 07:12:10 RouterName1 kernel: eth7: Link down >> > > May 20 07:12:10 RouterName1 kernel: net_ratelimit: 1228 callbacks >> > > suppressed >> > > May 20 07:12:10 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 07:12:16 RouterName1 kernel: last message repeated 9 times >> > > May 20 07:12:16 RouterName1 kernel: net_ratelimit: 312 callbacks >> suppressed >> > > May 20 07:12:16 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 07:12:22 RouterName1 kernel: last message repeated 9 times >> > > May 20 07:12:22 RouterName1 kernel: eth3: Link down >> > > May 20 07:12:23 RouterName1 kernel: net_ratelimit: 240 callbacks >> suppressed >> > > May 20 07:12:23 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 07:12:30 RouterName1 kernel: last message repeated 8 times >> > > May 20 07:12:30 RouterName1 kernel: eth1: Link down >> > > May 20 07:12:30 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 07:12:31 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 07:12:33 RouterName1 kernel: eth0: Link down >> > > May 20 07:12:33 RouterName1 kernel: nf_conntrack: table full, dropping >> > > packet >> > > May 20 07:13:05 RouterName1 kernel: last message repeated 2 times >> > > >> > > >> > > I have not had connection tracking table issues on larger networks >> with >> > > other vendors. Incluing my home network with ~100 routers and 3000 >> > > customers. We haven't seen connection tracking issues on his network >> > > with his ImageStreams, even when the ImageStreams were handling >> traffic >> > > for well over 500 clients in one bridge. I've run ImageStream, Cisco, >> > > MikroTik, and StarOS (blech) routers. None of them have given me the >> > > fits that the EdgeRouters have. >> > > >> > > The BGP speaking EdgeRouters which pass traffic for all customers have >> > > not been seen to have the connection tracking issue. I don't >> understand >> > > how devices taking a subset of the same traffic the BGP routers see >> can >> > > have issues while the BGP routers do not. >> > > >> > > Only the BGP speaking EdgeRouters do NAT for private address space. >> > > Customers all have public address space, so there is not much NAT >> > > traffic. The NATed traffic is primarily just NTP lookups and checks >> for >> > > updated software on Ubiquiti's site, including DNS lookups. >> > > >> > > What magic incantation am I missing? Maybe I'm just not looking at >> > > something correctly. >> > > >> > > Some of the time eating issues we've had have been the standard >> learning >> > > curve due to it being my first run with VyOS or derivitives. But a >> lot >> > > of time is just finding out the the WebUI can't actually do most of >> what >> > > needs doing in a real world ISP environment. Especially the things it >> > > claims to be able to do. Things like setting up DHCP servers to >> handle >> > > multiple subnets on one interface. Or OSPF setting up interfaces to >> > > use md5 key-id 1 which keeps it from talking to the other vendors who >> > > seem to all use md5 key-id 10. The key-id is not part of the >> interface >> > > configuration in the web-UI. A quick trip to the command line adding >> > > the key-id 10 makes the association work. Or you can dig through the >> > > config tree tab, which is actually more work. >> > > >> > > Only the conntrack table issue is a show stopper. The rest is merely >> > > annoying and makes me wish there wasn't a WebUI. Then I wouldn't be >> > > continually disappointed that I can't just hand it to the owner and >> say >> > > "this is where you go to do X in the WebUI." >> > > >> > > I went into setting up the EdgeRouters >> > > with an open mind looking forward to learning new toys. I am more and >> > > more wary of EdgeOS. EdgeOS is not really giving the owner the web >> > > manageability he was looking for anyway since so much of what an ISP >> > > needs to do cannot be done through the WebUI. >> > > >> > > -- >> > > Scott Lambert KC5MLE Unix >> SysAdmin >> > > lamb...@lambertfam.org <javascript:;> >> > > _______________________________________________ >> > > Ubnt_users mailing list >> > > Ubnt_users@wispa.org <javascript:;> >> > > http://lists.wispa.org/mailman/listinfo/ubnt_users >> > > >> > >> > >> > -- >> > Thanks, >> > Andrew Duey >> > Network Engineer, Widerange Broadband >> > Direct: 402-327-1101 >> >> > _______________________________________________ >> > Ubnt_users mailing list >> > Ubnt_users@wispa.org >> > http://lists.wispa.org/mailman/listinfo/ubnt_users >> >> >> -- >> Scott Lambert KC5MLE Unix >> SysAdmin >> lamb...@lambertfam.org >> _______________________________________________ >> Ubnt_users mailing list >> Ubnt_users@wispa.org >> http://lists.wispa.org/mailman/listinfo/ubnt_users >> > > > -- > Thanks, > Andrew Duey > Network Engineer, Widerange Broadband > Direct: 402-327-1101 > > > > _______________________________________________ > Ubnt_users mailing list > Ubnt_users@wispa.org > http://lists.wispa.org/mailman/listinfo/ubnt_users > >
_______________________________________________ Ubnt_users mailing list Ubnt_users@wispa.org http://lists.wispa.org/mailman/listinfo/ubnt_users
