Re: [AFMUG] strange outage

[Chuck McCown via AF]

Then just VLAN every AP back to your router.  This presumes your switches can 
handle VLANS.  You cannot have APs able to talk to each other through switches 
at towers or this will happen again.  

From: Jan-GAMs 
Sent: Friday, June 18, 2021 9:31 AM
To: af@af.afmug.com 
Subject: Re: [AFMUG] strange outage

We're under 100 subs, and static routing has been easy to monitor via the UISP. 
 Every CPE is displayed and easy to login to.  Any units on DHCP is a total 
PITA and I'd prefer to shoot the guy that started doing that as we can't find 
the user nor login to fix them, it's a truck roll which consumes hours instead 
of 5 minutes.  I would like to know more about setting up networks and finding 
the users once DHCP is deployed, since this is what our IT group is doing, 
drives me crazy.  I like to run a pro-active service dept instead of waiting 
for complaints.


Well we could replace the switches with routers but won't this reduce the total 
traffic available?  And once the traffic passes through several towers it gets 
reduced to not much available, more so than wide open links in bridge mode?


On 6/18/21 8:16 AM, Sam Lambie wrote:

  We had a flat network for a few years with the same setup as you in terms of 
network. Once the network grew to a certain size, broadcast storms would roll 
through often and it was almost impossible to track down the culprit without 
unplugging the gear and waiting for it to die down. We then switched to VLANs 
and that helped immensely, but was hard to manage as most everything was static 
routes and we had to remember where everything was routed. 
  Finally, we moved to routers at each tower, Natting the whole network and 
doing a whole slew of other things at the core side and so far for the past few 
years, it has been nice. 

  On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs  wrote:

I think this is beyond our present capability.  We have an edgerouter X 
where the network meets the internet and that's it.  There is only one OSPF, 
it's just one path with no other routes.  We have a switch at every tower that 
powers the APs and clients(CPE) that connect to APs.  We use UISP to monitor 
the network remotely.  Each CPE radio is a router but all are in "bridge" mode 
and we have different brands of routers inside the customer homes, non-ubnt 
devices are using dhcp.  We use one VLAN for management.  All customers are set 
to 20MBps for traffic control.


I couldn't find the guilty radio if there was one and the traffic being 
shown at the final uplink to the outside world would only pass about 0.1kbps 
using the built-in speedtest between it and the next closest link but the 
traffic monitor was showing peaks of about 6Mbps for total traffic.  I found 
nothing that could prove the traffic was real.

  There doesn't seem to be enough functions available in the CPEs to 
actively prevent this problem from happening again.  I'm not sure what you mean 
by "multicast"?  It makes sense to figure out a way to squelch it.


On 6/18/21 7:15 AM, Adam Moffett wrote:

  This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not 
sure how it handles multicast.  If everyone was in the same layer2 domain a 
heavy broadcast traffic could affect the whole system.  Maybe the customer 
moving 6-10mbps was malfunctioning and broadcasting something.

  In general it's safe to block all multicast and only allow it where you 
need to make OSPF connections.  Broadcast can be limited to 10kbps per customer 
with no issue.  The only broadcast they need to function is an ARP for their 
default gateway and a DHCP discover.  After initial discovery the DHCP traffic 
switches to unicast.  Not sure what tools ubnt gives you for filtering that, 
but ideally you'd block multicast and limit broadcast at every CPE.  




  On 6/18/2021 9:33 AM, Daniel White wrote:

Sounds like a broadcast storm to me.  What is the topology of your 
network?  Routers at each tower, VLANs, etc.?

Are you filtering multicast and broadcast traffic at the CPE/customer 
premises?


 Daniel White
Co-Founder 
phone: +1 (702) 470-2770
direct: +1 (702) 470-2766
   
 

  Jan-GAMsJune 17, 2021 at 23:47
  We had a strange outage on one of our networks yesterday.  At first 
we thought it was one customer.  The symptom was very low to non-existent 
internet traffic.  The complaint was my internet is not working! 

  Upon testing I found that the complaining customer had for a speed 
test about 0.14kbps for a speed to it's AP.  So I went to their AP and tested 
the speed back at them, it was about the same unusually slow speed.  Then I 
tested that AP to another AP and that speed was about the same slow speed.  So 
then I tested another customer and another and then ended up testing just about 
everyone in the whole network.  Everyone was opera

Re: [AFMUG] strange outage

[Chuck McCown via AF]

Yes, learned all the same lessons.  18 years ago.
Managed switches or routers at every tower.  Every single AP on its own VLAN or 
router port.  

From: Sam Lambie 
Sent: Friday, June 18, 2021 9:16 AM
To: AnimalFarm Microwave Users Group 
Subject: Re: [AFMUG] strange outage

We had a flat network for a few years with the same setup as you in terms of 
network. Once the network grew to a certain size, broadcast storms would roll 
through often and it was almost impossible to track down the culprit without 
unplugging the gear and waiting for it to die down. We then switched to VLANs 
and that helped immensely, but was hard to manage as most everything was static 
routes and we had to remember where everything was routed. 
Finally, we moved to routers at each tower, Natting the whole network and doing 
a whole slew of other things at the core side and so far for the past few 
years, it has been nice. 

On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs  wrote:

  I think this is beyond our present capability.  We have an edgerouter X where 
the network meets the internet and that's it.  There is only one OSPF, it's 
just one path with no other routes.  We have a switch at every tower that 
powers the APs and clients(CPE) that connect to APs.  We use UISP to monitor 
the network remotely.  Each CPE radio is a router but all are in "bridge" mode 
and we have different brands of routers inside the customer homes, non-ubnt 
devices are using dhcp.  We use one VLAN for management.  All customers are set 
to 20MBps for traffic control.


  I couldn't find the guilty radio if there was one and the traffic being shown 
at the final uplink to the outside world would only pass about 0.1kbps using 
the built-in speedtest between it and the next closest link but the traffic 
monitor was showing peaks of about 6Mbps for total traffic.  I found nothing 
that could prove the traffic was real.

There doesn't seem to be enough functions available in the CPEs to actively 
prevent this problem from happening again.  I'm not sure what you mean by 
"multicast"?  It makes sense to figure out a way to squelch it.


  On 6/18/21 7:15 AM, Adam Moffett wrote:

This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not sure 
how it handles multicast.  If everyone was in the same layer2 domain a heavy 
broadcast traffic could affect the whole system.  Maybe the customer moving 
6-10mbps was malfunctioning and broadcasting something.

In general it's safe to block all multicast and only allow it where you 
need to make OSPF connections.  Broadcast can be limited to 10kbps per customer 
with no issue.  The only broadcast they need to function is an ARP for their 
default gateway and a DHCP discover.  After initial discovery the DHCP traffic 
switches to unicast.  Not sure what tools ubnt gives you for filtering that, 
but ideally you'd block multicast and limit broadcast at every CPE.  




On 6/18/2021 9:33 AM, Daniel White wrote:

  Sounds like a broadcast storm to me.  What is the topology of your 
network?  Routers at each tower, VLANs, etc.?

  Are you filtering multicast and broadcast traffic at the CPE/customer 
premises?


   Daniel White
  Co-Founder 
  phone: +1 (702) 470-2770
  direct: +1 (702) 470-2766
 
   

Jan-GAMsJune 17, 2021 at 23:47
We had a strange outage on one of our networks yesterday.  At first we 
thought it was one customer.  The symptom was very low to non-existent internet 
traffic.  The complaint was my internet is not working! 

Upon testing I found that the complaining customer had for a speed test 
about 0.14kbps for a speed to it's AP.  So I went to their AP and tested the 
speed back at them, it was about the same unusually slow speed.  Then I tested 
that AP to another AP and that speed was about the same slow speed.  So then I 
tested another customer and another and then ended up testing just about 
everyone in the whole network.  Everyone was operating at an unusually slow 
speedtest to any other device of about 0.1kbps to 0kbps.  The whole network was 
down and yet the UISP was indicating everyone was up and operating with even 
some traffic in the 6-10 Mbps range which I'm sure was fake traffic as none of 
the devices tested would pass anything above a few kbps. 

A reboot of every device resolved the issue. 

Our gear is Ubiquiti and I'm wondering has anyone else using Ubiquiti 
been experiencing anything like what I just described? Is there a known cause? 






   

 
  -- 
  AF mailing list
  AF@af.afmug.com
  http://af.afmug.com/mailman/listinfo/af_af.afmug.com



-- 

-- 
Sam Lambie
Taosnet Wireless Tech.
575-758-7598 Office
www.Taosnet.com



-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
--

Re: [AFMUG] strange outage

[Carl Peterson]

We've gone full circle - Flat to fully routed to MPLS/VPLS over a routed
network back to flat.  You hit a scaling issue with routed networks as you
hit 10G and above, especially if you aren't using Mikrotik or other  low
cost routing.  Real carrier grade switching is a lot lower cost, lower
power, and much easier to manage.

Every customer has their own dedicated circuit (SVLAN.CVLAN).  The
corresponding interface on the BNG is dynamically created for the
subscriber with attributes out of radius.   Something like this isn't the
right answer at 100 customers but you should consider it or something like
it once you go north of a few k subs.

On Fri, Jun 18, 2021 at 10:32 AM Jan-GAMs  wrote:

> We're under 100 subs, and static routing has been easy to monitor via the
> UISP.  Every CPE is displayed and easy to login to.  Any units on DHCP is a
> total PITA and I'd prefer to shoot the guy that started doing that as we
> can't find the user nor login to fix them, it's a truck roll which consumes
> hours instead of 5 minutes.  I would like to know more about setting up
> networks and finding the users once DHCP is deployed, since this is what
> our IT group is doing, drives me crazy.  I like to run a pro-active service
> dept instead of waiting for complaints.
>
> Well we could replace the switches with routers but won't this reduce the
> total traffic available?  And once the traffic passes through several
> towers it gets reduced to not much available, more so than wide open links
> in bridge mode?
> On 6/18/21 8:16 AM, Sam Lambie wrote:
>
> We had a flat network for a few years with the same setup as you in terms
> of network. Once the network grew to a certain size, broadcast storms would
> roll through often and it was almost impossible to track down the culprit
> without unplugging the gear and waiting for it to die down. We then
> switched to VLANs and that helped immensely, but was hard to manage as most
> everything was static routes and we had to remember where everything was
> routed.
> Finally, we moved to routers at each tower, Natting the whole network and
> doing a whole slew of other things at the core side and so far for the past
> few years, it has been nice.
>
> On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs  wrote:
>
>> I think this is beyond our present capability.  We have an edgerouter X
>> where the network meets the internet and that's it.  There is only one
>> OSPF, it's just one path with no other routes.  We have a switch at every
>> tower that powers the APs and clients(CPE) that connect to APs.  We use
>> UISP to monitor the network remotely.  Each CPE radio is a router but all
>> are in "bridge" mode and we have different brands of routers inside the
>> customer homes, non-ubnt devices are using dhcp.  We use one VLAN for
>> management.  All customers are set to 20MBps for traffic control.
>>
>> I couldn't find the guilty radio if there was one and the traffic being
>> shown at the final uplink to the outside world would only pass about
>> 0.1kbps using the built-in speedtest between it and the next closest link
>> but the traffic monitor was showing peaks of about 6Mbps for total
>> traffic.  I found nothing that could prove the traffic was real.
>>
>>   There doesn't seem to be enough functions available in the CPEs to
>> actively prevent this problem from happening again.  I'm not sure what you
>> mean by "multicast"?  It makes sense to figure out a way to squelch it.
>> On 6/18/21 7:15 AM, Adam Moffett wrote:
>>
>> This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not
>> sure how it handles multicast.  If everyone was in the same layer2 domain a
>> heavy broadcast traffic could affect the whole system.  Maybe the customer
>> moving 6-10mbps was malfunctioning and broadcasting something.
>>
>> In general it's safe to block all multicast and only allow it where you
>> need to make OSPF connections.  Broadcast can be limited to 10kbps per
>> customer with no issue.  The only broadcast they need to function is an ARP
>> for their default gateway and a DHCP discover.  After initial discovery the
>> DHCP traffic switches to unicast.  Not sure what tools ubnt gives you for
>> filtering that, but ideally you'd block multicast and limit broadcast at
>> every CPE.
>>
>>
>> On 6/18/2021 9:33 AM, Daniel White wrote:
>>
>> Sounds like a broadcast storm to me.  What is the topology of your
>> network?  Routers at each tower, VLANs, etc.?
>>
>> Are you filtering multicast and broadcast traffic at the CPE/customer
>> premises?
>>
>> [image: photograph]
>> Daniel White
>> Co-Founder
>> phone: +1 (702) 470-2770
>> direct: +1 (702) 470-2766
>>
>> Jan-GAMs 
>> June 17, 2021 at 23:47
>> We had a strange outage on one of our networks yesterday.  At first we
>> thought it was one customer.  The symptom was very low to non-existent
>> internet traffic.  The complaint was my internet is not working!
>>
>> Upon testing I found that the complaining customer had for a speed test
>> about 

Re: [AFMUG] strange outage

[Adam Moffett]

No this is simply not true.

or the only nugget of truth is that it might take a beefier router 
to match the performance of a switch.


On 6/18/2021 11:31 AM, Jan-GAMs wrote:
Well we could replace the switches with routers but won't this reduce 
the total traffic available?  And once the traffic passes through 
several towers it gets reduced to not much available, more so than 
wide open links in bridge mode?


--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] strange outage

[Sam Lambie]

I have never heard of routers eating bandwidth between hops. Unless you are
thinking of mesh topology. Eww. Bridged transparent Point to point links
between towers, routers at each site. No bandwidth lost.

On Fri, Jun 18, 2021 at 9:33 AM Jan-GAMs  wrote:

> We're under 100 subs, and static routing has been easy to monitor via the
> UISP.  Every CPE is displayed and easy to login to.  Any units on DHCP is a
> total PITA and I'd prefer to shoot the guy that started doing that as we
> can't find the user nor login to fix them, it's a truck roll which consumes
> hours instead of 5 minutes.  I would like to know more about setting up
> networks and finding the users once DHCP is deployed, since this is what
> our IT group is doing, drives me crazy.  I like to run a pro-active service
> dept instead of waiting for complaints.
>
> Well we could replace the switches with routers but won't this reduce the
> total traffic available?  And once the traffic passes through several
> towers it gets reduced to not much available, more so than wide open links
> in bridge mode?
> On 6/18/21 8:16 AM, Sam Lambie wrote:
>
> We had a flat network for a few years with the same setup as you in terms
> of network. Once the network grew to a certain size, broadcast storms would
> roll through often and it was almost impossible to track down the culprit
> without unplugging the gear and waiting for it to die down. We then
> switched to VLANs and that helped immensely, but was hard to manage as most
> everything was static routes and we had to remember where everything was
> routed.
> Finally, we moved to routers at each tower, Natting the whole network and
> doing a whole slew of other things at the core side and so far for the past
> few years, it has been nice.
>
> On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs  wrote:
>
>> I think this is beyond our present capability.  We have an edgerouter X
>> where the network meets the internet and that's it.  There is only one
>> OSPF, it's just one path with no other routes.  We have a switch at every
>> tower that powers the APs and clients(CPE) that connect to APs.  We use
>> UISP to monitor the network remotely.  Each CPE radio is a router but all
>> are in "bridge" mode and we have different brands of routers inside the
>> customer homes, non-ubnt devices are using dhcp.  We use one VLAN for
>> management.  All customers are set to 20MBps for traffic control.
>>
>> I couldn't find the guilty radio if there was one and the traffic being
>> shown at the final uplink to the outside world would only pass about
>> 0.1kbps using the built-in speedtest between it and the next closest link
>> but the traffic monitor was showing peaks of about 6Mbps for total
>> traffic.  I found nothing that could prove the traffic was real.
>>
>>   There doesn't seem to be enough functions available in the CPEs to
>> actively prevent this problem from happening again.  I'm not sure what you
>> mean by "multicast"?  It makes sense to figure out a way to squelch it.
>> On 6/18/21 7:15 AM, Adam Moffett wrote:
>>
>> This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not
>> sure how it handles multicast.  If everyone was in the same layer2 domain a
>> heavy broadcast traffic could affect the whole system.  Maybe the customer
>> moving 6-10mbps was malfunctioning and broadcasting something.
>>
>> In general it's safe to block all multicast and only allow it where you
>> need to make OSPF connections.  Broadcast can be limited to 10kbps per
>> customer with no issue.  The only broadcast they need to function is an ARP
>> for their default gateway and a DHCP discover.  After initial discovery the
>> DHCP traffic switches to unicast.  Not sure what tools ubnt gives you for
>> filtering that, but ideally you'd block multicast and limit broadcast at
>> every CPE.
>>
>>
>> On 6/18/2021 9:33 AM, Daniel White wrote:
>>
>> Sounds like a broadcast storm to me.  What is the topology of your
>> network?  Routers at each tower, VLANs, etc.?
>>
>> Are you filtering multicast and broadcast traffic at the CPE/customer
>> premises?
>>
>> [image: photograph]
>> Daniel White
>> Co-Founder
>> phone: +1 (702) 470-2770
>> direct: +1 (702) 470-2766
>>
>> Jan-GAMs 
>> June 17, 2021 at 23:47
>> We had a strange outage on one of our networks yesterday.  At first we
>> thought it was one customer.  The symptom was very low to non-existent
>> internet traffic.  The complaint was my internet is not working!
>>
>> Upon testing I found that the complaining customer had for a speed test
>> about 0.14kbps for a speed to it's AP.  So I went to their AP and tested
>> the speed back at them, it was about the same unusually slow speed.  Then I
>> tested that AP to another AP and that speed was about the same slow speed.
>> So then I tested another customer and another and then ended up testing
>> just about everyone in the whole network.  Everyone was operating at an
>> unusually slow speedtest to any other device of about 0.1kbps to 

Re: [AFMUG] strange outage

[Jan-GAMs]

We're under 100 subs, and static routing has been easy to monitor via 
the UISP.  Every CPE is displayed and easy to login to.  Any units on 
DHCP is a total PITA and I'd prefer to shoot the guy that started doing 
that as we can't find the user nor login to fix them, it's a truck roll 
which consumes hours instead of 5 minutes.  I would like to know more 
about setting up networks and finding the users once DHCP is deployed, 
since this is what our IT group is doing, drives me crazy.  I like to 
run a pro-active service dept instead of waiting for complaints.


Well we could replace the switches with routers but won't this reduce 
the total traffic available?  And once the traffic passes through 
several towers it gets reduced to not much available, more so than wide 
open links in bridge mode?


On 6/18/21 8:16 AM, Sam Lambie wrote:
We had a flat network for a few years with the same setup as you in 
terms of network. Once the network grew to a certain size, broadcast 
storms would roll through often and it was almost impossible to track 
down the culprit without unplugging the gear and waiting for it to die 
down. We then switched to VLANs and that helped immensely, but was 
hard to manage as most everything was static routes and we had to 
remember where everything was routed.
Finally, we moved to routers at each tower, Natting the whole network 
and doing a whole slew of other things at the core side and so far for 
the past few years, it has been nice.


On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs > wrote:


I think this is beyond our present capability.  We have an
edgerouter X where the network meets the internet and that's it. 
There is only one OSPF, it's just one path with no other routes. 
We have a switch at every tower that powers the APs and
clients(CPE) that connect to APs. We use UISP to monitor the
network remotely.  Each CPE radio is a router but all are in
"bridge" mode and we have different brands of routers inside the
customer homes, non-ubnt devices are using dhcp.  We use one VLAN
for management.  All customers are set to 20MBps for traffic control.

I couldn't find the guilty radio if there was one and the traffic
being shown at the final uplink to the outside world would only
pass about 0.1kbps using the built-in speedtest between it and the
next closest link but the traffic monitor was showing peaks of
about 6Mbps for total traffic.  I found nothing that could prove
the traffic was real.

  There doesn't seem to be enough functions available in the CPEs
to actively prevent this problem from happening again.  I'm not
sure what you mean by "multicast"?  It makes sense to figure out a
way to squelch it.

On 6/18/21 7:15 AM, Adam Moffett wrote:


This is plausible.  I think ubnt sends broadcast traffic at
MCS0.  Not sure how it handles multicast.  If everyone was in the
same layer2 domain a heavy broadcast traffic could affect the
whole system.  Maybe the customer moving 6-10mbps was
malfunctioning and broadcasting something.

In general it's safe to block all multicast and only allow it
where you need to make OSPF connections. Broadcast can be limited
to 10kbps per customer with no issue.  The only broadcast they
need to function is an ARP for their default gateway and a DHCP
discover. After initial discovery the DHCP traffic switches to
unicast.  Not sure what tools ubnt gives you for filtering that,
but ideally you'd block multicast and limit broadcast at every CPE.


On 6/18/2021 9:33 AM, Daniel White wrote:

Sounds like a broadcast storm to me.  What is the topology of
your network?  Routers at each tower, VLANs, etc.?

Are you filtering multicast and broadcast traffic at the
CPE/customer premises?

photograph  
Daniel White
Co-Founder
phone: +1 (702) 470-2770
direct:+1 (702) 470-2766


Jan-GAMs 
June 17, 2021 at 23:47
We had a strange outage on one of our networks yesterday.  At
first we thought it was one customer.  The symptom was very low
to non-existent internet traffic.  The complaint was my
internet is not working!

Upon testing I found that the complaining customer had for a
speed test about 0.14kbps for a speed to it's AP.  So I went to
their AP and tested the speed back at them, it was about the
same unusually slow speed.  Then I tested that AP to another AP
and that speed was about the same slow speed.  So then I tested
another customer and another and then ended up testing just
about everyone in the whole network.  Everyone was operating at
an unusually slow speedtest to any other device of about
0.1kbps to 0kbps.  The whole network was down and yet the UISP
was indicating everyone was up and operating with even some
traffic in the 6-10 Mbps range which I'm sure was fake 

Re: [AFMUG] strange outage

[Sam Lambie]

We had a flat network for a few years with the same setup as you in terms
of network. Once the network grew to a certain size, broadcast storms would
roll through often and it was almost impossible to track down the culprit
without unplugging the gear and waiting for it to die down. We then
switched to VLANs and that helped immensely, but was hard to manage as most
everything was static routes and we had to remember where everything was
routed.
Finally, we moved to routers at each tower, Natting the whole network and
doing a whole slew of other things at the core side and so far for the past
few years, it has been nice.

On Fri, Jun 18, 2021 at 9:04 AM Jan-GAMs  wrote:

> I think this is beyond our present capability.  We have an edgerouter X
> where the network meets the internet and that's it.  There is only one
> OSPF, it's just one path with no other routes.  We have a switch at every
> tower that powers the APs and clients(CPE) that connect to APs.  We use
> UISP to monitor the network remotely.  Each CPE radio is a router but all
> are in "bridge" mode and we have different brands of routers inside the
> customer homes, non-ubnt devices are using dhcp.  We use one VLAN for
> management.  All customers are set to 20MBps for traffic control.
>
> I couldn't find the guilty radio if there was one and the traffic being
> shown at the final uplink to the outside world would only pass about
> 0.1kbps using the built-in speedtest between it and the next closest link
> but the traffic monitor was showing peaks of about 6Mbps for total
> traffic.  I found nothing that could prove the traffic was real.
>
>   There doesn't seem to be enough functions available in the CPEs to
> actively prevent this problem from happening again.  I'm not sure what you
> mean by "multicast"?  It makes sense to figure out a way to squelch it.
> On 6/18/21 7:15 AM, Adam Moffett wrote:
>
> This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not
> sure how it handles multicast.  If everyone was in the same layer2 domain a
> heavy broadcast traffic could affect the whole system.  Maybe the customer
> moving 6-10mbps was malfunctioning and broadcasting something.
>
> In general it's safe to block all multicast and only allow it where you
> need to make OSPF connections.  Broadcast can be limited to 10kbps per
> customer with no issue.  The only broadcast they need to function is an ARP
> for their default gateway and a DHCP discover.  After initial discovery the
> DHCP traffic switches to unicast.  Not sure what tools ubnt gives you for
> filtering that, but ideally you'd block multicast and limit broadcast at
> every CPE.
>
>
> On 6/18/2021 9:33 AM, Daniel White wrote:
>
> Sounds like a broadcast storm to me.  What is the topology of your
> network?  Routers at each tower, VLANs, etc.?
>
> Are you filtering multicast and broadcast traffic at the CPE/customer
> premises?
>
> [image: photograph]
> Daniel White
> Co-Founder
> phone: +1 (702) 470-2770
> direct: +1 (702) 470-2766
>
> Jan-GAMs 
> June 17, 2021 at 23:47
> We had a strange outage on one of our networks yesterday.  At first we
> thought it was one customer.  The symptom was very low to non-existent
> internet traffic.  The complaint was my internet is not working!
>
> Upon testing I found that the complaining customer had for a speed test
> about 0.14kbps for a speed to it's AP.  So I went to their AP and tested
> the speed back at them, it was about the same unusually slow speed.  Then I
> tested that AP to another AP and that speed was about the same slow speed.
> So then I tested another customer and another and then ended up testing
> just about everyone in the whole network.  Everyone was operating at an
> unusually slow speedtest to any other device of about 0.1kbps to 0kbps.
> The whole network was down and yet the UISP was indicating everyone was up
> and operating with even some traffic in the 6-10 Mbps range which I'm sure
> was fake traffic as none of the devices tested would pass anything above a
> few kbps.
>
> A reboot of every device resolved the issue.
>
> Our gear is Ubiquiti and I'm wondering has anyone else using Ubiquiti been
> experiencing anything like what I just described? Is there a known cause?
>
>
>
>
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>


-- 
-- 
*Sam Lambie*
Taosnet Wireless Tech.
575-758-7598 Office
www.Taosnet.com 
-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] strange outage

[Jan-GAMs]

I think this is beyond our present capability.  We have an edgerouter X 
where the network meets the internet and that's it. There is only one 
OSPF, it's just one path with no other routes. We have a switch at every 
tower that powers the APs and clients(CPE) that connect to APs.  We use 
UISP to monitor the network remotely.  Each CPE radio is a router but 
all are in "bridge" mode and we have different brands of routers inside 
the customer homes, non-ubnt devices are using dhcp.  We use one VLAN 
for management.  All customers are set to 20MBps for traffic control.


I couldn't find the guilty radio if there was one and the traffic being 
shown at the final uplink to the outside world would only pass about 
0.1kbps using the built-in speedtest between it and the next closest 
link but the traffic monitor was showing peaks of about 6Mbps for total 
traffic.  I found nothing that could prove the traffic was real.


  There doesn't seem to be enough functions available in the CPEs to 
actively prevent this problem from happening again.  I'm not sure what 
you mean by "multicast"?  It makes sense to figure out a way to squelch it.


On 6/18/21 7:15 AM, Adam Moffett wrote:


This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not 
sure how it handles multicast.  If everyone was in the same layer2 
domain a heavy broadcast traffic could affect the whole system.  Maybe 
the customer moving 6-10mbps was malfunctioning and broadcasting 
something.


In general it's safe to block all multicast and only allow it where 
you need to make OSPF connections.  Broadcast can be limited to 10kbps 
per customer with no issue.  The only broadcast they need to function 
is an ARP for their default gateway and a DHCP discover.  After 
initial discovery the DHCP traffic switches to unicast.  Not sure what 
tools ubnt gives you for filtering that, but ideally you'd block 
multicast and limit broadcast at every CPE.



On 6/18/2021 9:33 AM, Daniel White wrote:
Sounds like a broadcast storm to me.  What is the topology of your 
network? Routers at each tower, VLANs, etc.?


Are you filtering multicast and broadcast traffic at the CPE/customer 
premises?


photograph  
Daniel White
Co-Founder
phone: +1 (702) 470-2770
direct:+1 (702) 470-2766


Jan-GAMs 
June 17, 2021 at 23:47
We had a strange outage on one of our networks yesterday.  At first 
we thought it was one customer.  The symptom was very low to 
non-existent internet traffic.  The complaint was my internet is not 
working!


Upon testing I found that the complaining customer had for a speed 
test about 0.14kbps for a speed to it's AP.  So I went to their AP 
and tested the speed back at them, it was about the same unusually 
slow speed.  Then I tested that AP to another AP and that speed was 
about the same slow speed.  So then I tested another customer and 
another and then ended up testing just about everyone in the whole 
network.  Everyone was operating at an unusually slow speedtest to 
any other device of about 0.1kbps to 0kbps. The whole network was 
down and yet the UISP was indicating everyone was up and operating 
with even some traffic in the 6-10 Mbps range which I'm sure was 
fake traffic as none of the devices tested would pass anything above 
a few kbps.


A reboot of every device resolved the issue.

Our gear is Ubiquiti and I'm wondering has anyone else using 
Ubiquiti been experiencing anything like what I just described? Is 
there a known cause?








-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] strange outage

[Adam Moffett]

This is plausible.  I think ubnt sends broadcast traffic at MCS0.  Not 
sure how it handles multicast.  If everyone was in the same layer2 
domain a heavy broadcast traffic could affect the whole system.  Maybe 
the customer moving 6-10mbps was malfunctioning and broadcasting something.


In general it's safe to block all multicast and only allow it where you 
need to make OSPF connections.  Broadcast can be limited to 10kbps per 
customer with no issue.  The only broadcast they need to function is an 
ARP for their default gateway and a DHCP discover.  After initial 
discovery the DHCP traffic switches to unicast.  Not sure what tools 
ubnt gives you for filtering that, but ideally you'd block multicast and 
limit broadcast at every CPE.



On 6/18/2021 9:33 AM, Daniel White wrote:
Sounds like a broadcast storm to me.  What is the topology of your 
network? Routers at each tower, VLANs, etc.?


Are you filtering multicast and broadcast traffic at the CPE/customer 
premises?


photograph  
Daniel White
Co-Founder
phone: +1 (702) 470-2770
direct:+1 (702) 470-2766


Jan-GAMs 
June 17, 2021 at 23:47
We had a strange outage on one of our networks yesterday.  At first 
we thought it was one customer.  The symptom was very low to 
non-existent internet traffic.  The complaint was my internet is not 
working!


Upon testing I found that the complaining customer had for a speed 
test about 0.14kbps for a speed to it's AP.  So I went to their AP 
and tested the speed back at them, it was about the same unusually 
slow speed.  Then I tested that AP to another AP and that speed was 
about the same slow speed.  So then I tested another customer and 
another and then ended up testing just about everyone in the whole 
network.  Everyone was operating at an unusually slow speedtest to 
any other device of about 0.1kbps to 0kbps.  The whole network was 
down and yet the UISP was indicating everyone was up and operating 
with even some traffic in the 6-10 Mbps range which I'm sure was fake 
traffic as none of the devices tested would pass anything above a few 
kbps.


A reboot of every device resolved the issue.

Our gear is Ubiquiti and I'm wondering has anyone else using Ubiquiti 
been experiencing anything like what I just described? Is there a 
known cause?






-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com

Re: [AFMUG] strange outage

[Daniel White]

Sounds like a broadcast storm to me.  What is the topology of your
network?  Routers at each tower, VLANs, etc.?

Are you filtering multicast and broadcast traffic at the CPE/customer
premises?

photograph  
Daniel White
Co-Founder
phone: +1 (702) 470-2770
direct:+1 (702) 470-2766

> Jan-GAMs 
> June 17, 2021 at 23:47
> We had a strange outage on one of our networks yesterday.  At first we
> thought it was one customer.  The symptom was very low to non-existent
> internet traffic.  The complaint was my internet is not working!
>
> Upon testing I found that the complaining customer had for a speed
> test about 0.14kbps for a speed to it's AP.  So I went to their AP and
> tested the speed back at them, it was about the same unusually slow
> speed.  Then I tested that AP to another AP and that speed was about
> the same slow speed.  So then I tested another customer and another
> and then ended up testing just about everyone in the whole network. 
> Everyone was operating at an unusually slow speedtest to any other
> device of about 0.1kbps to 0kbps.  The whole network was down and yet
> the UISP was indicating everyone was up and operating with even some
> traffic in the 6-10 Mbps range which I'm sure was fake traffic as none
> of the devices tested would pass anything above a few kbps.
>
> A reboot of every device resolved the issue.
>
> Our gear is Ubiquiti and I'm wondering has anyone else using Ubiquiti
> been experiencing anything like what I just described? Is there a
> known cause?
>
>

-- 
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com