Scott, To answer your questions, I do not have any junipers or real Cisco units in the network. I've worked with a few smaller Cisco units but nothing of even the same caliber. I only have a handful of the EdgeRouter Pro's at each site I support. For the ISP network we have a pair of EdgeRouter Pro's at the core and then have Mikrotik at all the tower sites. We had been using Vyatta for years before Brocade bought them and burnt all bridges with me (what a terrible, worthless, corporate greed driven company).
--Andrew On Friday, June 10, 2016, Scott Lambert <lamb...@lambertfam.org> wrote: > Thank you for the reply. I would not keep attempting to make the WebUI > work, except the guy paying me to set these up specifically does not > want to type more than he has to. He's not much of a typist. > > If i could find an equivalent of RouterOS's "/interface monitor-traffic > ether1" or cisco's "show interfaces GigabitEthernet 0/3", I would never > load EdgeOS's heavy WebUI. > > I just compared the connection tracking timeouts between EdgeOS and > RouterOS. > > It looks to me like RouterOS is at least three times as aggressive > about getting rid of connections which were likely to be malicious > traffic. It also looks to me as though RouterOS autoscales the size of > the connection tracking table when it gets "too full"; whatever "too > full" means. > > The minimum size I've seen is 88088 entries, the maximum on my network > is somewhere over 472,000 entries on a box which has lots of firewall > rules, does NAT, and passes 800Mbps during peak hours for more than 1500 > users. That box is using less than 1GB of RAM. Maybe I'll just set all > of the EdgeRouters to use similar settings to that CCR1036. > > Just out of curiosity, how many EdgeRouters do you have deployed? > > Do you use anything between the Juniper and the EdgeRouter Pro? > > I'm at around 100 RouterOS routers in the ISP network. Several more > at client businesses. In ISP networks, I've only used about 10 > ImageStreams and a few dozen Ciscos. No Junipers, yet. > > The Ciscos are hands down the reliability champs, followed by > ImageStream, then RouterOS, then the EdgeRouters, in my experience so > far. > > Like you said, it's hard to justify the big brands when MikroTik and > Ubiquiti have such low cost options in the CCR1009s and the EdgeRouter > Pros. > > I better get back to looking up how to setup GRE tunnels across the > Internet and run OSPF across them as a backup path for when the network > gets partitioned internally on the EdgeRouters. > > On Thu, Jun 09, 2016 at 11:05:03PM -0500, Andrew Duey wrote: > > Scott, > > > > We have been running the EdgeRouter Pro's for at least a year now and ran > > Vyatta for a couple years before that. After reading your frustrations > > here's a couple observations: > > > > 1) We do NO NAT in our Edgerouters now, but when we did in Vyatta we had > > the same conntrack issues. We found you could increase the table size > > dramatically and that fixed our problems until we could un-NAT > everything. > > I would recommend not doing NAT on those units at all but if you have to > > you can increase the table sizes to help. > > > > 2) Ours have been crazy stable (given the very low cost of the units). > > > > 3) The web gui is only useful for viewing real-time network > information. I > > do like that you can see real-time traffic counts, see how routing > changes > > take effect, and see an outage in a heartbeat but the web guy is for > > information only. I would STRONGLY recommend that in an ISP environment > > that ALL configuration be done at the command line. I found it very > > friendly and easy to use, and there used to be TONS of peer Vyatta > support > > out there (Brocade killed a lot of it, but there's still a lot left in > > other places). > > > > 4) While UBNT's direct support is crap on a good day, I've found that > their > > forums have 90% of the answers I'm looking for and the rest can be gotten > > with my googlefu and searching for Vyatta examples. > > > > While the $400 EdgeRouter Pro is no Juniper MX160 I've found that they > are > > amazing little boxes given their price point. We make a point to always > > keep at least one spare on hand and run them in pairs anyway but so far > > haven't needed to worry about them too much (knock on wood). > > > > Best of luck with them > > --Andrew Duey > > > > On Thursday, June 9, 2016, Scott Lambert <lamb...@lambertfam.org > <javascript:;>> wrote: > > > > > Warning, this message has not been well editted. I don't have time to > > > go back and get it to try to get the tone to match what is intended. I > > > have limitted time onsite during which to try to fix issues. However > > > "rant"y this sounds, it should sound like frustration rather than > anger. > > > > > > I am helping an ISP owner who is trying to transition from one big > > > bridge to a mostly routed network. He had ImageStream routers at three > > > points, two points were BGP upstreams where there is not really enough > > > bandwidth between the sites to fail over gracefully. We are working on > > > that too. > > > > > > Last summer, he started running into devices falling over because the > > > bridge table was full, so we tried putting an EdgeRouter at one site > > > toward the long end of the network to get about 400 devices off the > main > > > bridge as a quick bandaid. It fell over about the time he got an hour > > > away from the site. We found conntrack issues in the log. We didn't > > > have any firewall rules. We rebooted and it ran fine, for an hour or > > > so. Then more conntrack issues. We quickly configured a spare 100Mbps > > > ImageStream and had no more issues. 100Mbps isn't really a long term > > > option but was okay at the time. > > > > > > In October, I came on-site and we built and installed another > EdgeRouter > > > PoE because we were half afraid of the original hardware. This time > > > v1.7.0 was available and we got it running without blowing up. At > least > > > for the week I was on-site. We also installed six other EdgeRouters > > > to replace the two BGP speaking ImageStreams and to segment four other > > > hub points in the network. Yea! no more bridge loops without turning > > > toughswitch ports on and off. OSPF fails over properly. Everything > > > looked okay while I was onsite. > > > > > > Since then, there have been a few times when no traffic would pass > > > through one of those hub sites and he would have to powercycle > > > everything. > > > > > > Then I happened to be on-site for the AirOS MF worm outbreak. Two of > > > the hub sites repeatedly stopped passing traffic while we were trying > > > to clean up. Being onsite I was able to be patient and persistent and > > > eventually get into the router at the site which was offline to view > the > > > log. The logs said the connection tracking table was full. Sometimes > > > my connection to the router would continue to work long enough to issue > > > a reboot command after seeing the logs. Sometimes I would have to be > > > patient and persistent to get back into the router to issue the reboot. > > > I tried to blame it on the AirOS MF worm traffic. > > > > > > We replaced his last ImageStream router with an EdgeRouter Pro the day > > > before I left after cleaning up the AirOS MF worm. We put 1.8.0 on > > > it and didn't expect any issues. Two hours after it started passing > > > traffic, it overflowed the conntrack tables. We heard from customers > at > > > 7 am that it had been down since 1:30 am. He moved the cables back to > > > the still powered Imagestream and all was well for the next month. I > > > came back and looked at the log on the still running EdgeRouter Pro and > > > it has nothing but "nf_conntrack: table full, dropping packet" errors > > > from 1:17 to 07:13. > > > > > > Shouldn't it have recovered at some point during the six hours? The > > > rest of the network was fine during this time. Connection tracking had > > > no issues on the BGP router which handles this traffic + the other 4 > > > times this number of clients traffic, there were no issues with the two > > > hub routers which pass traffic in the ring from the BGP router to this > > > router. > > > > > > #Powered up, cables connected: > > > Jan 1 00:24:15 RouterName1 OSPF[906]: OSPF-6: Message-digest-key > key-id > > > 1 enter Passive mode > > > Jan 1 00:25:29 RouterName1 OSPF[906]: OSPF-6: Message-digest-key > key-id > > > 1 enter Passive mode > > > May 19 22:56:41 RouterName1 OSPF[906]: OSPF-4: Detected router with > > > duplicate router ID10.11.100.2 in area 0.0.0.0 > > > > > > # I am disappointed that there were no link up messages as we moved the > > > # cables to this device. The last OSPF speaking cable was moved after > > > # the above log message. The EdgeRouter uses the same router ID as the > > > # Imagestream. So the above message was expected. > > > > > > May 20 00:17:01 RouterName1 rsyslogd: set SCM_CREDENTIALS failed on > > > '/dev/log': Protocol not avai > > > lable > > > May 20 01:17:26 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 01:17:31 RouterName1 kernel: last message repeated 9 times > > > May 20 01:17:31 RouterName1 kernel: net_ratelimit: 87 callbacks > suppressed > > > May 20 01:17:31 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 01:17:36 RouterName1 kernel: last message repeated 9 times > > > May 20 01:17:36 RouterName1 kernel: net_ratelimit: 73 callbacks > suppressed > > > May 20 01:17:36 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 01:17:41 RouterName1 kernel: last message repeated 9 times > > > May 20 01:17:41 RouterName1 kernel: net_ratelimit: 118 callbacks > suppressed > > > May 20 01:17:41 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 01:17:46 RouterName1 kernel: last message repeated 9 times > > > May 20 01:17:46 RouterName1 kernel: net_ratelimit: 262 callbacks > suppressed > > > May 20 01:17:46 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 01:17:51 RouterName1 kernel: last message repeated 9 times > > > > > > #Last ethernet cord being pulled: > > > May 20 07:12:10 RouterName1 kernel: eth7: Link down > > > May 20 07:12:10 RouterName1 kernel: net_ratelimit: 1228 callbacks > > > suppressed > > > May 20 07:12:10 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 07:12:16 RouterName1 kernel: last message repeated 9 times > > > May 20 07:12:16 RouterName1 kernel: net_ratelimit: 312 callbacks > suppressed > > > May 20 07:12:16 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 07:12:22 RouterName1 kernel: last message repeated 9 times > > > May 20 07:12:22 RouterName1 kernel: eth3: Link down > > > May 20 07:12:23 RouterName1 kernel: net_ratelimit: 240 callbacks > suppressed > > > May 20 07:12:23 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 07:12:30 RouterName1 kernel: last message repeated 8 times > > > May 20 07:12:30 RouterName1 kernel: eth1: Link down > > > May 20 07:12:30 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 07:12:31 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 07:12:33 RouterName1 kernel: eth0: Link down > > > May 20 07:12:33 RouterName1 kernel: nf_conntrack: table full, dropping > > > packet > > > May 20 07:13:05 RouterName1 kernel: last message repeated 2 times > > > > > > > > > I have not had connection tracking table issues on larger networks with > > > other vendors. Incluing my home network with ~100 routers and 3000 > > > customers. We haven't seen connection tracking issues on his network > > > with his ImageStreams, even when the ImageStreams were handling traffic > > > for well over 500 clients in one bridge. I've run ImageStream, Cisco, > > > MikroTik, and StarOS (blech) routers. None of them have given me the > > > fits that the EdgeRouters have. > > > > > > The BGP speaking EdgeRouters which pass traffic for all customers have > > > not been seen to have the connection tracking issue. I don't > understand > > > how devices taking a subset of the same traffic the BGP routers see can > > > have issues while the BGP routers do not. > > > > > > Only the BGP speaking EdgeRouters do NAT for private address space. > > > Customers all have public address space, so there is not much NAT > > > traffic. The NATed traffic is primarily just NTP lookups and checks > for > > > updated software on Ubiquiti's site, including DNS lookups. > > > > > > What magic incantation am I missing? Maybe I'm just not looking at > > > something correctly. > > > > > > Some of the time eating issues we've had have been the standard > learning > > > curve due to it being my first run with VyOS or derivitives. But a lot > > > of time is just finding out the the WebUI can't actually do most of > what > > > needs doing in a real world ISP environment. Especially the things it > > > claims to be able to do. Things like setting up DHCP servers to handle > > > multiple subnets on one interface. Or OSPF setting up interfaces to > > > use md5 key-id 1 which keeps it from talking to the other vendors who > > > seem to all use md5 key-id 10. The key-id is not part of the interface > > > configuration in the web-UI. A quick trip to the command line adding > > > the key-id 10 makes the association work. Or you can dig through the > > > config tree tab, which is actually more work. > > > > > > Only the conntrack table issue is a show stopper. The rest is merely > > > annoying and makes me wish there wasn't a WebUI. Then I wouldn't be > > > continually disappointed that I can't just hand it to the owner and say > > > "this is where you go to do X in the WebUI." > > > > > > I went into setting up the EdgeRouters > > > with an open mind looking forward to learning new toys. I am more and > > > more wary of EdgeOS. EdgeOS is not really giving the owner the web > > > manageability he was looking for anyway since so much of what an ISP > > > needs to do cannot be done through the WebUI. > > > > > > -- > > > Scott Lambert KC5MLE Unix > SysAdmin > > > lamb...@lambertfam.org <javascript:;> <javascript:;> > > > _______________________________________________ > > > Ubnt_users mailing list > > > Ubnt_users@wispa.org <javascript:;> <javascript:;> > > > http://lists.wispa.org/mailman/listinfo/ubnt_users > > > > > > > > > -- > > Thanks, > > Andrew Duey > > Network Engineer, Widerange Broadband > > Direct: 402-327-1101 > > > _______________________________________________ > > Ubnt_users mailing list > > Ubnt_users@wispa.org <javascript:;> > > http://lists.wispa.org/mailman/listinfo/ubnt_users > > > -- > Scott Lambert KC5MLE Unix SysAdmin > lamb...@lambertfam.org <javascript:;> > _______________________________________________ > Ubnt_users mailing list > Ubnt_users@wispa.org <javascript:;> > http://lists.wispa.org/mailman/listinfo/ubnt_users > -- Thanks, Andrew Duey Network Engineer, Widerange Broadband Direct: 402-327-1101
_______________________________________________ Ubnt_users mailing list Ubnt_users@wispa.org http://lists.wispa.org/mailman/listinfo/ubnt_users
