Yes, learned all the same lessons. 18 years ago.
Managed switches or routers at every tower. Every single AP on its own VLAN or
router port.
From: Sam Lambie
Sent: Friday, June 18, 2021 9:16 AM
To: AnimalFarm Microwave Users Group
Subject: Re: [AFMUG] strange outage
We had a flat network for a few years with the same setup as you in terms of
network. Once the network grew to a certain size, broadcast storms would roll
through often and it was almost impossible to track down the culprit without
unplugging the gear and waiting for it to die down. We then switched to VLANs
and that helped immensely, but was hard to manage as most everything was static
routes and we had to remember where everything was routed.
Finally, we moved to routers at each tower, Natting the whole network and doing
a whole slew of other things at the core side and so far for the past few
years, it has been nice.
On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs <j.vank...@grnacres.net> wrote:
I think this is beyond our present capability. We have an edgerouter X where
the network meets the internet and that's it. There is only one OSPF, it's
just one path with no other routes. We have a switch at every tower that
powers the APs and clients(CPE) that connect to APs. We use UISP to monitor
the network remotely. Each CPE radio is a router but all are in "bridge" mode
and we have different brands of routers inside the customer homes, non-ubnt
devices are using dhcp. We use one VLAN for management. All customers are set
to 20MBps for traffic control.
I couldn't find the guilty radio if there was one and the traffic being shown
at the final uplink to the outside world would only pass about 0.1kbps using
the built-in speedtest between it and the next closest link but the traffic
monitor was showing peaks of about 6Mbps for total traffic. I found nothing
that could prove the traffic was real.
There doesn't seem to be enough functions available in the CPEs to actively
prevent this problem from happening again. I'm not sure what you mean by
"multicast"? It makes sense to figure out a way to squelch it.
On 6/18/21 7:15 AM, Adam Moffett wrote:
This is plausible. I think ubnt sends broadcast traffic at MCS0. Not sure
how it handles multicast. If everyone was in the same layer2 domain a heavy
broadcast traffic could affect the whole system. Maybe the customer moving
6-10mbps was malfunctioning and broadcasting something.
In general it's safe to block all multicast and only allow it where you
need to make OSPF connections. Broadcast can be limited to 10kbps per customer
with no issue. The only broadcast they need to function is an ARP for their
default gateway and a DHCP discover. After initial discovery the DHCP traffic
switches to unicast. Not sure what tools ubnt gives you for filtering that,
but ideally you'd block multicast and limit broadcast at every CPE.
On 6/18/2021 9:33 AM, Daniel White wrote:
Sounds like a broadcast storm to me. What is the topology of your
network? Routers at each tower, VLANs, etc.?
Are you filtering multicast and broadcast traffic at the CPE/customer
premises?
Daniel White
Co-Founder
phone: +1 (702) 470-2770
direct: +1 (702) 470-2766
Jan-GAMsJune 17, 2021 at 23:47
We had a strange outage on one of our networks yesterday. At first we
thought it was one customer. The symptom was very low to non-existent internet
traffic. The complaint was my internet is not working!
Upon testing I found that the complaining customer had for a speed test
about 0.14kbps for a speed to it's AP. So I went to their AP and tested the
speed back at them, it was about the same unusually slow speed. Then I tested
that AP to another AP and that speed was about the same slow speed. So then I
tested another customer and another and then ended up testing just about
everyone in the whole network. Everyone was operating at an unusually slow
speedtest to any other device of about 0.1kbps to 0kbps. The whole network was
down and yet the UISP was indicating everyone was up and operating with even
some traffic in the 6-10 Mbps range which I'm sure was fake traffic as none of
the devices tested would pass anything above a few kbps.
A reboot of every device resolved the issue.
Our gear is Ubiquiti and I'm wondering has anyone else using Ubiquiti
been experiencing anything like what I just described? Is there a known cause?
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
--
Sam Lambie
Taosnet Wireless Tech.
575-758-7598 Office
www.Taosnet.com
--------------------------------------------------------------------------------
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
