[WISPA] Taking Mikrotik down
Brett, I'm impressed with your knowledge of Mikrotik programming so I wanted to ask you this. Last week and further back about four times a week we had a cascading crash of our bridged network whereas the LAN side of the Mikrotik Backhauls would crash presumably from traffic. Wireshark showed some anomalies such as IPv6 traffic, some ICMP sneaking through the filters, a random STP Cisco and some TCP flooding but really nothing that should take down so many radios, a simple reboot fixed the problem and it didn't happen again for a day to several days later. Friday we changed out three Mikrotik backhauls and AP's with Ubiquity gear and upgraded our Bandwidth manager enhancing its rules as well. Today we're having the same attack as before but now it's not taking down the system, Our bandwidth monitor is pegged on incoming traffic and outgoing traffic at 176% of normal (we normally peak at 99% download 30% up) but no radio's are going down, our system latency at the affected tower is 300ms and we're getting intermittent down alarms. Its great because we have the first chance to go customer by customer trying to find the source but I guess I'm asking if you have any ideas how to find or filter this problem? We think the source is comin Thanks, Forbes WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Taking Mikrotik down
On your ubnt equipment - did you enable stp ? using a syslog -? might show a good amount of info if so On Sep 13, 2010, at 3:30 PM, Forbes Mercy wrote: > Brett, I'm impressed with your knowledge of Mikrotik programming so I > wanted to ask you this. Last week and further back about four times a > week we had a cascading crash of our bridged network whereas the LAN > side of the Mikrotik Backhauls would crash presumably from traffic. > Wireshark showed some anomalies such as IPv6 traffic, some ICMP sneaking > through the filters, a random STP Cisco and some TCP flooding but really > nothing that should take down so many radios, a simple reboot fixed the > problem and it didn't happen again for a day to several days later. > > Friday we changed out three Mikrotik backhauls and AP's with Ubiquity > gear and upgraded our Bandwidth manager enhancing its rules as well. > Today we're having the same attack as before but now it's not taking > down the system, Our bandwidth monitor is pegged on incoming traffic and > outgoing traffic at 176% of normal (we normally peak at 99% download 30% > up) but no radio's are going down, our system latency at the affected > tower is 300ms and we're getting intermittent down alarms. Its great > because we have the first chance to go customer by customer trying to > find the source but I guess I'm asking if you have any ideas how to find > or filter this problem? We think the source is comin > > Thanks, > Forbes > > > > WISPA Wants You! Join today! > http://signup.wispa.org/ > > > WISPA Wireless List: wireless@wispa.org > > Subscribe/Unsubscribe: > http://lists.wispa.org/mailman/listinfo/wireless > > Archives: http://lists.wispa.org/pipermail/wireless/ _ Glenn Kelley | Principle | HostMedic |www.HostMedic.com Email: gl...@hostmedic.com Pplease don't print this e-mail unless you really need to. WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Taking Mikrotik down
On Mon, 2010-09-13 at 12:30 -0700, Forbes Mercy wrote: > Brett, Not sure who Brett is, but... > Wireshark showed some anomalies such as IPv6 traffic There is more IPV6 traffic on the network than most people realize. I've got captures from about 40 networks dating back to about a year and a half ago that nearly ALL have IPv6 traffic (tunneled over v4 of course). > Friday we changed out three Mikrotik backhauls and AP's with Ubiquity > gear and upgraded our Bandwidth manager enhancing its rules as well. > Today we're having the same attack as before but now it's not taking > down the system, Our bandwidth monitor is pegged on incoming traffic and > outgoing traffic at 176% of normal (we normally peak at 99% download 30% > up) but no radio's are going down, our system latency at the affected > tower is 300ms and we're getting intermittent down alarms. Its great > because we have the first chance to go customer by customer trying to > find the source but I guess I'm asking if you have any ideas how to find > or filter this problem? We think the source is comin Not sure what the rest of this sentence was going to be, but I may be able to offer some "quickie" checks to do. 1. Run torch on the interface(s) in question 2. Sort by source or destination IP (depending on which one is your customer IP range). 3. Look for patterns such as: a. one IP making many connections to the same IP on different ports (port scanner) b. one IP making many connections to many IPs on the same port (virus) c. one IP making many connections to many IPs on different ports (likely to be a torrent or other P2P) 4. If you don't see any of the above patterns, sort by bandwidth (tx then rx rates) and look to see if it is just one user consuming an inordinate amount of bandwidth This should give you a starting point anyway. -- * Butch Evans * Professional Network Consultation* * http://www.butchevans.com/* Network Engineering * * http://store.wispgear.net/* Wired or Wireless Networks * * http://blog.butchevans.com/ * ImageStream, Mikrotik and MORE! * WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Taking Mikrotik down
We inherited a bridged network (bridged EoIP tunnels over routing w/OSPF) that had similar problems. It turned out to be caused by Belkin (and similar) routers that resend packets they don't think they should have received (or don't know what to do with) back out their WAN interface. If you have two of these routers on the same ethernet segment...boom. It loops until one of your backhauls fails and breaks the cycle. The reason I think this is similar to your case is that maybe the Ubiquiti backhauls you installed have a lower PPS capability, and in a way are protecting your router from the onslaught. One way to test this theory is to set the horizon to the same value on all of your bridge ports facing customers at all of your aggregation points. This will prevent the bridge from forwarding traffic between customers. It worked for us while trying to isolate the problem because each tower has it's own EoIP tunnel terminating on a central router. So, we could control what gets forwarded where in one location. If your network is bridged from core to edge, then this would be harder (not impossible) to implement. If you do this, though, customers on the same IP subnet won't be able to reach each other. Regards, -Kristian On Mon, 2010-09-13 at 12:30 -0700, Forbes Mercy wrote: > Brett, I'm impressed with your knowledge of Mikrotik programming so I > wanted to ask you this. Last week and further back about four times a > week we had a cascading crash of our bridged network whereas the LAN > side of the Mikrotik Backhauls would crash presumably from traffic. > Wireshark showed some anomalies such as IPv6 traffic, some ICMP sneaking > through the filters, a random STP Cisco and some TCP flooding but really > nothing that should take down so many radios, a simple reboot fixed the > problem and it didn't happen again for a day to several days later. > > Friday we changed out three Mikrotik backhauls and AP's with Ubiquity > gear and upgraded our Bandwidth manager enhancing its rules as well. > Today we're having the same attack as before but now it's not taking > down the system, Our bandwidth monitor is pegged on incoming traffic and > outgoing traffic at 176% of normal (we normally peak at 99% download 30% > up) but no radio's are going down, our system latency at the affected > tower is 300ms and we're getting intermittent down alarms. Its great > because we have the first chance to go customer by customer trying to > find the source but I guess I'm asking if you have any ideas how to find > or filter this problem? We think the source is comin > > Thanks, > Forbes > > > > WISPA Wants You! Join today! > http://signup.wispa.org/ > > > WISPA Wireless List: wireless@wispa.org > > Subscribe/Unsubscribe: > http://lists.wispa.org/mailman/listinfo/wireless > > Archives: http://lists.wispa.org/pipermail/wireless/ > WISPA Wants You! Join today! http://signup.wispa.org/ WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
