Re: plea for comcast/sprint handoff debug help

[Randy Bush]

i'll see your blog post and raise you a peer reviewed academic paper and
two rfcs :)

in dnssec, we want to move from the old mandatory to implement (mti) rsa
signatures to the more modern ecdsa.

how would the world work out if i fielded a validating dns cache server
which *implemented* rsa, because it is mti, but chose not to actually
*use* it for validation on odd numbered wednesdays because of my
religious belief that ecdsa is superior?

perhaps go over to your unbound siblings and discuss this analog.

but thanks for your help in getting jtk's imc paper accepted. :)

randy

Re: DE-CIX - Wednesday 11:00am - 1:30pm Eastern

[Thomas King]

Hi Matt,

 

please open a ticket with supp...@de-cix.net and/or call +1-212-796-6914 so 
that we can have a direct communication with you. We will work with you on 
resolving the issue right away.

 

Best regards,

Thomas


-- 

Dr. Thomas King

Chief Technology Officer (CTO)

 

DE-CIX Management GmbH | Lindleystraße 12 | 60314 Frankfurt am Main | Germany | 
www.de-cix.net |

Phone +49 69 1730902 87 | Mobile +49 175 1161428 | Fax +49 69 4056 2716 | 
thomas.k...@de-cix.net |

Geschaeftsfuehrer Harald A. Summa and Sebastian Seifert | Registergericht AG 
Koeln HRB 51135

 

DE-CIX 25th anniversary: Without you the Internet would not be the same!

Join us on the journey at https://withoutyou.de-cix.net

 

 

From: NANOG  on behalf of Matt 
Harris 
Date: Thursday, 29. October 2020 at 23:19
To: Matt Hoppes 
Cc: North American Network Operators' Group 
Subject: Re: DE-CIX - Wednesday 11:00am - 1:30pm Eastern

 

Matt Harris​
|

Infrastructure Lead Engineer

816‑256‑5446
|

Direct

Looking for something?
Helpdesk Portal
|

Email Support

|

Billing Portal

We build and deliver end‑to‑end IT solutions.
On Thu, Oct 29, 2020 at 11:10 AM Matt Hoppes 
 wrote:

Hi Ed,
Thank you for that clarification.

On 10/29/20 12:05 PM, Ed dAgostino wrote:
> All,
> 
> 
> For clarification, DE-CIX New York operates over a dozen switches as 
> part of our local switch fabric.   ONE of our switches malfunctioned for 
> about a two hour period  prior to rebooting and that caused problems for 
> customer networks connected to that switch during that period.   All 
> other switches were not impacted and the DE-CIX New York exchange did 
> not go down.
> 
> 
> Ed d'Agostino

 

It looks as though something's going on again right now. I just lost sessions 
to one of the two route servers, and some but not all of my other peers on 
DE-CIX NY. 

 



smime.p7s
Description: S/MIME cryptographic signature

Re: DE-CIX - Wednesday 11:00am - 1:30pm Eastern

[Matt Harris]

On Thu, Oct 29, 2020 at 11:10 AM Matt Hoppes <
mattli...@rivervalleyinternet.net> wrote:

> Hi Ed,
> Thank you for that clarification.
>
> On 10/29/20 12:05 PM, Ed dAgostino wrote:
> > All,
> >
> >
> > For clarification, DE-CIX New York operates over a dozen switches as
> > part of our local switch fabric.   ONE of our switches malfunctioned for
> > about a two hour period  prior to rebooting and that caused problems for
> > customer networks connected to that switch during that period.   All
> > other switches were not impacted and the DE-CIX New York exchange did
> > not go down.
> >
> >
> > Ed d'Agostino


It looks as though something's going on again right now. I just lost
sessions to one of the two route servers, and some but not all of my other
peers on DE-CIX NY.

Matt Harris|Infrastructure Lead Engineer
816-256-5446|Direct
Looking for something?
Helpdesk Portal|Email Support|Billing Portal
We build and deliver end-to-end IT solutions.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Aaron C. de Bruyn via NANOG]

Sorry--accidental premature send.

On Thu, Oct 29, 2020 at 12:54 PM Brielle  wrote:

> Updates are from same link as above, and there's new builds based on
> their new OS that integrates a bunch of separate controllers if you
> don't mind beta...
>

With most companies I wouldn't mind.
But with UniFi, Alpha is Beta, and Beta is launched into production.


> > I have a UDM Pro that's useless because the WAN port seems to have a
> > manufacturing defect.
>
> Its actually a software bug - I had it too on releases around 1.8.0.
> It's actually fixed in 1.8.2-5 and later.
>

That's what I saw originally too.  It kept it sitting on the shelf for
months.
It's fixed in their beta software, but not on one of my appliances.


> One thing I will note - I've had only one device out of hundreds go bad
> in the last 10 years of using their products.  Usually, when you have
> lots of failing hardware 'for some reason', its a good idea to look into
> the reason 'why?'.  Could be bad grounds, could be dirty power...


Their switches and WAPs are pretty solid for me.  I have about 150 of each
deployed with only 2 or 3 failures.

I've had electricians check power in locations where things died and they
find no issues.  Plus the sites use double-conversion UPS units, so I
highly doubt it's power issues.

-A

Re: Apple Catalina Appears to Introduce Massive Jitter

[Aaron C. de Bruyn via NANOG]

On Thu, Oct 29, 2020 at 12:54 PM Brielle  wrote:

> On 10/29/2020 1:42 PM, Aaron C. de Bruyn via NANOG wrote:
> > I have an old CloudKey that mysteriously doesn't seem to be getting
> > updates anymore.
>
>
> https://community.ui.com/releases/UniFi-Cloud-Key-Firmware-1-1-13/733dfc55-b61b-483b-afc1-77d7f2c1e032
>
> 4 months ago, if you want to stick strictly to stable...  More recent if
> you run a beta release.


Not  to drag this out any further, but that was well after I stopped using
it because it hadn't had updates for nearly a year.

Same with the other issues mentioned.  They spent a good 8-12 months not
communicating with customers, not addressing issues, and not releasing
firmware while they developed their new UniFi OS.

-A

Recommendation to update RPKI validators

[Job Snijders]

Hi all,

About eight months ago I discovered a number of issues in the validation
procedure of most RPKI validator softwares (including the RIPE NCC
Validator, Routinator, and OctoRPKI). The impact of improper
verification of Manifests (and associated aspects of the X.509 system)
in the RPKI can have rather dramatic effects in today's Internet routing
landscape. When handling a manifest, make sure everything is accounted
for!

The mitigation guidance is at present is very simple: just make sure all
deployed RPKI validators are updated to the latest version.

Going forward I hope our industry as a whole will be able to respond
faster to issues of this type. A write-up with examples and details is
available here: http://sobornost.net/~job/manifest_handling_issue.txt

Thank you to all involved who helped fix & progress this issue.

Kind regards,

Job

Re: plea for comcast/sprint handoff debug help

[Randy Bush]

>>> tl;dr:
>>> 
>>> comcast: does your 50.242.151.5 westin router receive the announcement
>>> of 147.28.0.0/20 from sprint's westin router 144.232.9.61?
>> 
>> tl;dr: diagnosed by comcast.  see our short paper to be presented at imc
>>   tomorrow https://archive.psg.com/200927.imc-rp.pdf
>> 
>> lesson: route origin relying party software may cause as much damage as
>>  it ameliorates
>> 
>> randy
> 
> To clarify this for the readers here: there is an ongoing research
> experiment where connectivity to the RRDP and rsync endpoints of
> several RPKI publication servers is being purposely enabled and
> disabled for prolonged periods of time. This is perfectly fine of
> course.
> 
> While the resulting paper presented at IMC is certainly interesting,
> having relying party software fall back to rsync when RRDP is
> unavailable is not a requirement specified in any RFC, as the paper
> seems to suggest. In fact, we argue that it's actually a bad idea to
> do so:
> 
> https://blog.nlnetlabs.nl/why-routinator-doesnt-fall-back-to-rsync/
> 
> We're interested to hear views on this from both an operational and
> security perspective.

in fact,  has found your bug.  if you find an http
server, but it is not serving the new and not-required rrdp protocol, it
does not then use the mandatory to implement rsync.

randy

Re: plea for comcast/sprint handoff debug help

[Alex Band]

> On 28 Oct 2020, at 16:58, Randy Bush  wrote:
> 
>> tl;dr:
>> 
>> comcast: does your 50.242.151.5 westin router receive the announcement
>> of 147.28.0.0/20 from sprint's westin router 144.232.9.61?
> 
> tl;dr: diagnosed by comcast.  see our short paper to be presented at imc
>   tomorrow https://archive.psg.com/200927.imc-rp.pdf
> 
> lesson: route origin relying party software may cause as much damage as
>   it ameliorates
> 
> randy

To clarify this for the readers here: there is an ongoing research experiment 
where connectivity to the RRDP and rsync endpoints of several RPKI publication 
servers is being purposely enabled and disabled for prolonged periods of time. 
This is perfectly fine of course.

While the resulting paper presented at IMC is certainly interesting, having 
relying party software fall back to rsync when RRDP is unavailable is not a 
requirement specified in any RFC, as the paper seems to suggest. In fact, we 
argue that it's actually a bad idea to do so:

https://blog.nlnetlabs.nl/why-routinator-doesnt-fall-back-to-rsync/

We're interested to hear views on this from both an operational and security 
perspective.

-Alex

Re: Apple Catalina Appears to Introduce Massive Jitter

[Brielle]

On 10/29/2020 1:42 PM, Aaron C. de Bruyn via NANOG wrote:
I have an old CloudKey that mysteriously doesn't seem to be getting 
updates anymore.


https://community.ui.com/releases/UniFi-Cloud-Key-Firmware-1-1-13/733dfc55-b61b-483b-afc1-77d7f2c1e032

4 months ago, if you want to stick strictly to stable...  More recent if 
you run a beta release.



I have an old CloudKey Gen2 Plus that hasn't received updates in about a 
year but the HDD died 8 months in.


Updates are from same link as above, and there's new builds based on 
their new OS that integrates a bunch of separate controllers if you 
don't mind beta...


Not sure about hard disk dying.


I have a UDM Pro that's useless because the WAN port seems to have a 
manufacturing defect.



Its actually a software bug - I had it too on releases around 1.8.0. 
It's actually fixed in 1.8.2-5 and later.



The other stuff, i can't really comment on.  But, I figured I'd correct 
several statements of what I had knowledge on.  :-)


One thing I will note - I've had only one device out of hundreds go bad 
in the last 10 years of using their products.  Usually, when you have 
lots of failing hardware 'for some reason', its a good idea to look into 
the reason 'why?'.  Could be bad grounds, could be dirty power...


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Apple Catalina Appears to Introduce Massive Jitter

[Aaron C. de Bruyn via NANOG]

On Thu, Oct 29, 2020 at 12:22 PM Peter Beckman  wrote:

> I'll take all of your Unifi gear, PM me for an address. :-)
>

I'd send it your way in a heartbeat, but you wouldn't get much use out of
it.
I have an old CloudKey that mysteriously doesn't seem to be getting updates
anymore.
I have an old CloudKey Gen2 Plus that hasn't received updates in about a
year but the HDD died 8 months in.
I have a UDM Pro that's useless because the WAN port seems to have a
manufacturing defect.
I have a 4 old UVC-G3 cameras that died at about 13 months (not eligible
for warranty repair)
I have 3 UVC-G3-Flex cameras that have bad SD cards that I fortunately
bought from a vendor instead of UniFi directly and they just said "here are
replacements, don't bother shipping that crap back--toss them".
I have a US-24-250 and a US-48-500 switch that have dead power supplies.
One was DOA, the other died about 3 months in.  UniFi won't fix it, and I'm
not going to pay for shipping labels to fix their mistakes.  (I paid for a
working switch, and I got a dead switch--why should I pay more to get what
I ordered?)
I have a dead VIEWPORT and two that 'stutter' badly when displaying 4
cameras and they reboot several times per hour.
I have a partially melted UA-HUB.

Fortunately I switched to a vendor in the last 6 months that charges about
5% less than the list price and they actually replace stuff at no cost.
Dealing with Ubiquity directly is a nightmare.

There are some quirky things about Unifi that can be annoying, but it is
> mostly around common stuff like running a DNS Caching server on the
> Security Gateway or force-pushing a DDNS update.
>

Quirky things:  the lack of customer service, the lack of communication
with customers, the buggy software, releasing hardware to production that
is completely unusable for 6 months (UDM Pro), etc...

Anyways, not to get too down on them, their wireless gear and switches are
pretty wonderful.  I've never had a WAP die on me, and only a small handful
of switches...but the cost is hard to beat.
There has been talk in the forums about various Apple gear having problems,
and I think a recent update (possibly still in beta) appears to have fixed
it.  I don't own any Apple gear, so I can't confirm.

-A

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mel Beckman]

I also have all Ubiquiti  stuff at home, and I’ve deployed it in large 
installations of up to 100 APs. Beyond that it seems to hit some communication 
bottleneck in its spectrum allocation protocols, so I usually go to Aruba or 
Ruckus for networks larger than that.

Ubiquiti is pretty reliable, but you have to stay away from the latest release, 
just like anything. Its SDN implementation is quite impressive for what it 
does, especially for the fact that you get it for free (i’m talking to you, 
Cisco DNA :-). The only major downside is that it’s all Ubiquiti all the time: 
they don’t interoperate with anything else.

-mel via cell

> On Oct 29, 2020, at 12:24 PM, Peter Beckman  wrote:
> 
> I'd need more data than your anecdotal experience with a POE device to
> throw out my Unifi gear and ban the company. But I'm dealing with 2
> devices: a Security Gateway and a single Access Point (plus the Controller
> software running on my Mac).
> 
> There are some quirky things about Unifi that can be annoying, but it is
> mostly around common stuff like running a DNS Caching server on the
> Security Gateway or force-pushing a DDNS update.
> 
> It's been way better than the ad-hoc varied brand of network I was running
> before, and I get to see and manage a lot more as well, quite reliably.
> 
> We all know that hardware failures happen, and you definitely had a bad
> experience, and that sucks.
> 
> I'll take all of your Unifi gear, PM me for an address. :-)
> 
> Beckman
> 
>> On Thu, 29 Oct 2020, Aaron C. de Bruyn via NANOG wrote:
>> 
>>> On Thu, Oct 29, 2020 at 5:43 AM Jared Mauch  wrote:
>>> 
>>>I have all UBNT at home for wireless and periodically have some
>>> random
>>> issues which I can't explain, but for the most part have things tuned to
>>> ensure
>>> there's little to no interference.
>>> 
>> 
>> All UBNT at home?  Ouch.
>> 
>> They're on my banned list after one of their POE devices caught on fire
>> after being in service for 11 months.
>> Then they went round and round for a week saying they weren't going to pay
>> for a shipping label.  I wasn't going to pay for one because I didn't want
>> their gear back.
>> 
>> Finally someone with a bit of common sense sent a shipping label so they
>> could figure out why it caught on fire.
>> They ended up sending a replacement back that was obviously used.  Instead
>> of letting it go to waste, I installed it.
>> It died two weeks later.  When I contacted them, they said the original
>> purchase was over a year ago so they wouldn't RMA it.
>> 
>> Then a second device (plugged into an entirely different switch in a
>> different building) started smoking and emitting an electrical smell.  I
>> pulled all of them and tossed them in the dumpster.
>> 
>> They are an absolutely atrocious company to deal with.  I'm betting some
>> day real soon they'll be sued into oblivion when their crap burns down
>> someone's home or office building.
>> 
>> Friends don't let friends buy UniFi.
>> 
>> -A
>> 
> 
> ---
> Peter Beckman  Internet Guy
> beck...@angryox.com http://www.angryox.com/
> ---

Re: Apple Catalina Appears to Introduce Massive Jitter

[Peter Beckman]

I'd need more data than your anecdotal experience with a POE device to
throw out my Unifi gear and ban the company. But I'm dealing with 2
devices: a Security Gateway and a single Access Point (plus the Controller
software running on my Mac).

There are some quirky things about Unifi that can be annoying, but it is
mostly around common stuff like running a DNS Caching server on the
Security Gateway or force-pushing a DDNS update.

It's been way better than the ad-hoc varied brand of network I was running
before, and I get to see and manage a lot more as well, quite reliably.

We all know that hardware failures happen, and you definitely had a bad
experience, and that sucks.

I'll take all of your Unifi gear, PM me for an address. :-)

Beckman

On Thu, 29 Oct 2020, Aaron C. de Bruyn via NANOG wrote:


On Thu, Oct 29, 2020 at 5:43 AM Jared Mauch  wrote:


I have all UBNT at home for wireless and periodically have some
random
issues which I can't explain, but for the most part have things tuned to
ensure
there's little to no interference.



All UBNT at home?  Ouch.

They're on my banned list after one of their POE devices caught on fire
after being in service for 11 months.
Then they went round and round for a week saying they weren't going to pay
for a shipping label.  I wasn't going to pay for one because I didn't want
their gear back.

Finally someone with a bit of common sense sent a shipping label so they
could figure out why it caught on fire.
They ended up sending a replacement back that was obviously used.  Instead
of letting it go to waste, I installed it.
It died two weeks later.  When I contacted them, they said the original
purchase was over a year ago so they wouldn't RMA it.

Then a second device (plugged into an entirely different switch in a
different building) started smoking and emitting an electrical smell.  I
pulled all of them and tossed them in the dumpster.

They are an absolutely atrocious company to deal with.  I'm betting some
day real soon they'll be sued into oblivion when their crap burns down
someone's home or office building.

Friends don't let friends buy UniFi.

-A



---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

Re: Apple Catalina Appears to Introduce Massive Jitter

[Aaron C. de Bruyn via NANOG]

On Thu, Oct 29, 2020 at 5:43 AM Jared Mauch  wrote:

> I have all UBNT at home for wireless and periodically have some
> random
> issues which I can't explain, but for the most part have things tuned to
> ensure
> there's little to no interference.
>

All UBNT at home?  Ouch.

They're on my banned list after one of their POE devices caught on fire
after being in service for 11 months.
Then they went round and round for a week saying they weren't going to pay
for a shipping label.  I wasn't going to pay for one because I didn't want
their gear back.

Finally someone with a bit of common sense sent a shipping label so they
could figure out why it caught on fire.
They ended up sending a replacement back that was obviously used.  Instead
of letting it go to waste, I installed it.
It died two weeks later.  When I contacted them, they said the original
purchase was over a year ago so they wouldn't RMA it.

Then a second device (plugged into an entirely different switch in a
different building) started smoking and emitting an electrical smell.  I
pulled all of them and tossed them in the dumpster.

They are an absolutely atrocious company to deal with.  I'm betting some
day real soon they'll be sued into oblivion when their crap burns down
someone's home or office building.

Friends don't let friends buy UniFi.

-A

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 19:38, colin johnston wrote:


Be careful using Apple wireless diagnostic package, uses a lot of /var/tmp 
space on a small Macbook air 128ssd


Mine cost me 300MB.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 19:24, colin johnston wrote:

This does seem to be solved with the checksum disable below, or at least pings 
down to sub 10ms on Mac book air with Catalina beta 10.15.6, why aim performs 
far better I don’t know. I tried to introduce load after cksum disable and it 
did not see ping spikes as before


Didn't work for me, sadly. I still had times ending up in the 235ms 
range after making the changes.


Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[colin johnston]

Be careful using Apple wireless diagnostic package, uses a lot of /var/tmp 
space on a small Macbook air 128ssd

Col

 
> On 29 Oct 2020, at 17:24, colin johnston  wrote:
> 
> This does seem to be solved with the checksum disable below, or at least 
> pings down to sub 10ms on Mac book air with Catalina beta 10.15.6, why aim 
> performs far better I don’t know. I tried to introduce load after cksum 
> disable and it did not see ping spikes as before
> 
> How do we now explain to Apple to fix ?
> 
> Col
> 
> 
>> On 29 Oct 2020, at 14:08, J. Hellenthal via NANOG  wrote:
>> 
>> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. 
>> Not diagnosing further because I could not find a reason for it fast enough 
>> and not sure if it really had an impact at the moment…. but could you try 
>> the following 
>> 
>> 
>> sudo sysctl net.link.generic.system.hwcksum_tx=0
>> sudo sysctl net.link.generic.system.hwcksum_rx=0
>> sudo ifconfig en0 -rxcsum
>> 
>> 
>> in reverse … to restore the settings 
>> 
>> sudo sysctl net.link.generic.system.hwcksum_tx=1
>> sudo sysctl net.link.generic.system.hwcksum_rx=1
>> sudo ifconfig en0 rxcsum
>> 
>> 
>> If you have some specific tests to run I would be willing to run them here 
>> on Big Sur with the same laptop but I have nothing now that runs Catalina
>> 
>> 
>> Wireshark used to in Catalina rack up cksum errors a lot while these were 
>> all at their defaults.
>> 
>> 
>> 
>>> On Oct 29, 2020, at 08:23, Mark Tinka  wrote:
>>> 
>>> 
>>> 
>>> On 10/29/20 15:04, Cory Sell wrote:
>>> 
 Might be worth disabling each AP to see if there's one out there having an 
 issue playing nice with the MacBook. Also try different combinations of 
 two APs working together. It's possible the MacBook is flip flopping 
 because the power levels are fighting each other.
>>> 
>>> Tested all that, as well as dropping Tx power levels on each of the AP's to 
>>> Low so that there isn't any power coming from any other AP (despite being 
>>> quite far, already).
>>> 
>>> And to confirm, when the laptop locks into an AP, it doesn't try to join 
>>> another one. When in range, power is very good (between -37dB and -52dB). 
>>> When I walk away, that AP becomes too far (as bad as -80dB), but the next 
>>> one close by is far better (same good values as before) and laptop connects 
>>> and sticks to that.
>>> 
>>> Again, only impacts Catalina. No other Apple device, or the Windows PC that 
>>> is on the same WLAN.
>>> 
>>> 
 Does the Mac have this issue at your local coffee shop or another 
 establishment with Wi-Fi? You can try to rule out the AirPort card in the 
 Mac itself.
>>> 
>>> Never tried, I generally work from home. If I'm out, it's faster to tether 
>>> to my 4G service rather than any public wi-fi.
>>> 
>>> Mark.
>> 
>> 
>> -- 
>> 
>> J. Hellenthal
>> 
>> The fact that there's a highway to Hell but only a stairway to Heaven says a 
>> lot about anticipated traffic volume.
>> 
>> 
>> 
>> 
>> 
>> 
> 

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 19:27, Randy Bush wrote:


you only *think* you turned off location services. as they are a vital
component of providing a good user experience ...

:(


That was an honest, lingering thought.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Randy Bush]

> Most folk from various fora suggested Location Services were to
> blame. I turned all of mine off, no joy.

you only *think* you turned off location services.  as they are a vital
component of providing a good user experience ...

:(

Re: Apple Catalina Appears to Introduce Massive Jitter

[colin johnston]

This does seem to be solved with the checksum disable below, or at least pings 
down to sub 10ms on Mac book air with Catalina beta 10.15.6, why aim performs 
far better I don’t know. I tried to introduce load after cksum disable and it 
did not see ping spikes as before

How do we now explain to Apple to fix ?

Col


> On 29 Oct 2020, at 14:08, J. Hellenthal via NANOG  wrote:
> 
> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. 
> Not diagnosing further because I could not find a reason for it fast enough 
> and not sure if it really had an impact at the moment…. but could you try the 
> following 
> 
> 
> sudo sysctl net.link.generic.system.hwcksum_tx=0
> sudo sysctl net.link.generic.system.hwcksum_rx=0
> sudo ifconfig en0 -rxcsum
> 
> 
> in reverse … to restore the settings 
> 
> sudo sysctl net.link.generic.system.hwcksum_tx=1
> sudo sysctl net.link.generic.system.hwcksum_rx=1
> sudo ifconfig en0 rxcsum
> 
> 
> If you have some specific tests to run I would be willing to run them here on 
> Big Sur with the same laptop but I have nothing now that runs Catalina
> 
> 
> Wireshark used to in Catalina rack up cksum errors a lot while these were all 
> at their defaults.
> 
> 
> 
>> On Oct 29, 2020, at 08:23, Mark Tinka  wrote:
>> 
>> 
>> 
>> On 10/29/20 15:04, Cory Sell wrote:
>> 
>>> Might be worth disabling each AP to see if there's one out there having an 
>>> issue playing nice with the MacBook. Also try different combinations of two 
>>> APs working together. It's possible the MacBook is flip flopping because 
>>> the power levels are fighting each other.
>> 
>> Tested all that, as well as dropping Tx power levels on each of the AP's to 
>> Low so that there isn't any power coming from any other AP (despite being 
>> quite far, already).
>> 
>> And to confirm, when the laptop locks into an AP, it doesn't try to join 
>> another one. When in range, power is very good (between -37dB and -52dB). 
>> When I walk away, that AP becomes too far (as bad as -80dB), but the next 
>> one close by is far better (same good values as before) and laptop connects 
>> and sticks to that.
>> 
>> Again, only impacts Catalina. No other Apple device, or the Windows PC that 
>> is on the same WLAN.
>> 
>> 
>>> Does the Mac have this issue at your local coffee shop or another 
>>> establishment with Wi-Fi? You can try to rule out the AirPort card in the 
>>> Mac itself.
>> 
>> Never tried, I generally work from home. If I'm out, it's faster to tether 
>> to my 4G service rather than any public wi-fi.
>> 
>> Mark.
> 
> 
> -- 
> 
> J. Hellenthal
> 
> The fact that there's a highway to Hell but only a stairway to Heaven says a 
> lot about anticipated traffic volume.
> 
> 
> 
> 
> 
> 

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 19:12, colin johnston wrote:


Hey Mark,
Good shout with debug, same issue seen on MacBook Air with Catalina 
10.15.6 beta, pings upto 150ms seen

iMac with Sierra zero jitter and usually sub 1m pings
Now need to find out why, I never noticed as wife using the MacBook Air :(
I cant yet update to big sur since need lots of sad space, need to 
cutdown on university docs me thinks


Thanks, Col.

So confirmed that Big Sur has the same problem, both on newer and older 
Mac's.


Issue seems to have surfaced somewhere around Mojave.

Most folk from various fora suggested Location Services were to blame. I 
turned all of mine off, no joy.


A friend of mine in Malaysia noticed an improvement when he disabled 
Handoff between the Mac and all his other iCloud-enabled devices. But 
that didn't work for me.


Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[colin johnston]

Hey Mark,
Good shout with debug, same issue seen on MacBook Air with Catalina 10.15.6 
beta, pings upto 150ms seen
iMac with Sierra zero jitter and usually sub 1m pings
Now need to find out why, I never noticed as wife using the MacBook Air :(
I cant yet update to big sur since need lots of sad space, need to cutdown on 
university docs me thinks

Col

> On 29 Oct 2020, at 12:07, Mark Tinka  wrote:
> 
> Hi all.
> 
> I've been on High Sierra for several years now due to a limitation with an 
> app that couldn't deal with Apple's latest rounds of system permissions since 
> Mojave. Eventually, I gave up on waiting for them to fix it and upgraded my 
> older Butterfly keyboard laptop to Catalina 4 weeks ago.
> 
> At the same time, I picked up the new Magic keyboard laptop 2 weeks ago which 
> came with Catalina.
> 
> Over the past week, I've been troubleshooting a massive jitter issue on 
> Catalina, just between itself and my home router. For control, I have a 
> Windows PC (tower-top) using a wireless adapter to connect to my home 
> network. That has no jitter at all.
> 
> I have noticed as much as 300ms+ jitter on Catalina.
> 
> I then asked a few friends around the world to run tests for me on their own 
> Catalina installations to their local router over wi-fi, and the results are 
> the same. Jitter so high that what should be a 1ms - 5ms latency can (for a 
> short period) jump to 200ms+, 300ms+, 400ms+.
> 
> On the off-chance that it is an issue with the new wireless chips on the 
> later MacBook models, one of my friends tested the same on a 2013 MacBook Pro 
> running a beta version of Big Sur. Same story!
> 
> Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro 
> running Catalina, also had the same issue.
> 
> A Google search suggests that this is some known issue since Mojave, to do 
> with Location Services, and some other apps, in a non-deterministic way:
> 
> 
> https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router
>  
> 
> 
> For me, even after disabling all or some Location Services features, the 
> problem remains.
> 
> Is anyone else seeing this on their Catalina Mac's while on wi-fi? If so, 
> does anyone know what's going on here?
> 
> Ideally, this wouldn't matter if it was just a cosmetic issue - but I do 
> actually see physical impact to performance of network access to/from the 
> laptop, which has all the hallmarks of high jitter and/or packet loss. 
> 
> An app like Zoom, which can display network performance data for a session in 
> real-time, does indicate nominal packet loss for audio and video on this 
> device, while other devices on the same WLAN are happy.
> 
> Thoughts?
> 
> Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 18:05, Blake Hudson wrote:

On the latest Catalina 10.15.7 from a MacBook Air (early 2014) via 
WiFi to Google Wifi Mesh router (only a single unit network):


Over 2.4Ghz through 3 interior walls:
--- 192.168.86.1 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.440/15.078/213.538/30.541 ms

Over 5Ghz through 1 door:
--- 192.168.86.1 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.557/24.614/137.233/39.015 ms


This is what I'm talking about. Why would you have such massive jitter 
on wi-fi to your local router?


You're hitting 213ms on 2.4GHz and 137ms on 5GHz. This doesn't seem odd 
to you?


Never had this issue pre-Catalina (I understand it may have been 
introduced in Mojave, but I came from High Sierra).



This is a real world test with several iPhones, 2.4GHz WiFi only 
cameras, and an iPad being used for zoom (remote school) while running 
these tests.


Same for me. There are tons of devices on my WLAN, and only the one 
running Catalina behaves this way.


No drama for the rest.


If you were not aware, on a Mac you can hold the Option key while 
clicking on the WiFi icon to see more options as well as more 
information about your current WiFi connection, including the realtime 
Tx rate, channel, and when your Mac is searching for new networks 
(mine does this every few seconds, it seems).


Yes, I am aware about the on-board macOS wi-fi diagnostics, and the good 
ol' trusted "airport -s". But those aren't enough.


I ended up spending money on NetSpot to see what's going on. Nothing 
there either. All I can tell is that Catalina is the problem; how 
exactly, is not yet very clear.



From an experience perspective, all applications seem to work fine for 
us. Including uploading to YouTube (a regular event as my spouse is a 
teacher) and Zoom/Teams/FaceTime/your teleconference app of choice 
(used by all of us).


So even though apps like Zoom report mild packet loss while on wi-fi 
(0.1% - 0.10%), it's not a train smash.


In general, everything works fine. But it doesn't feel 100%.

For the Youtube upload, I've found it to be temperamental, but I'm 
making it work for the time being.


Mark.

Re: DE-CIX - Wednesday 11:00am - 1:30pm Eastern

[Matt Hoppes]

Hi Ed,
Thank you for that clarification.

On 10/29/20 12:05 PM, Ed dAgostino wrote:

All,


For clarification, DE-CIX New York operates over a dozen switches as 
part of our local switch fabric.   ONE of our switches malfunctioned for 
about a two hour period  prior to rebooting and that caused problems for 
customer networks connected to that switch during that period.   All 
other switches were not impacted and the DE-CIX New York exchange did 
not go down.



Ed d'Agostino


*From:* NANOG  on 
behalf of Matt Hoppes 

*Sent:* Thursday, October 29, 2020 11:29:03 AM
*To:* North American Network Operators' Group; 
outages-discuss...@outages.org

*Subject:* DE-CIX - Wednesday 11:00am - 1:30pm Eastern
Does anyone here know any more about what crashed in the DE-CIX NYC
exchange yesterday between approximately 11am and 1:30pm?

Between those times I'm told the Nokia switching fabric locked up and
ultimately rebooted.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Blake Hudson]

On the latest Catalina 10.15.7 from a MacBook Air (early 2014) via WiFi 
to Google Wifi Mesh router (only a single unit network):


Over 2.4Ghz through 3 interior walls:
--- 192.168.86.1 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.440/15.078/213.538/30.541 ms

Over 5Ghz through 1 door:
--- 192.168.86.1 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.557/24.614/137.233/39.015 ms

This is a real world test with several iPhones, 2.4GHz WiFi only 
cameras, and an iPad being used for zoom (remote school) while running 
these tests. If you were not aware, on a Mac you can hold the Option key 
while clicking on the WiFi icon to see more options as well as more 
information about your current WiFi connection, including the realtime 
Tx rate, channel, and when your Mac is searching for new networks (mine 
does this every few seconds, it seems).


From an experience perspective, all applications seem to work fine for 
us. Including uploading to YouTube (a regular event as my spouse is a 
teacher) and Zoom/Teams/FaceTime/your teleconference app of choice (used 
by all of us).


--B


On 10/29/2020 7:28 AM, Mark Tinka wrote:

Ah yes, an example of what I am seeing:


Marks-MacBook-Pro.local (172.16.0.239) 2020-10-29T14:28:27+0200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.16.0.254 0.8%   126    3.9  34.7   2.5 232.1  54.9

Mark.

On 10/29/20 14:07, Mark Tinka wrote:

Hi all.

I've been on High Sierra for several years now due to a limitation 
with an app that couldn't deal with Apple's latest rounds of system 
permissions since Mojave. Eventually, I gave up on waiting for them 
to fix it and upgraded my older Butterfly keyboard laptop to Catalina 
4 weeks ago.


At the same time, I picked up the new Magic keyboard laptop 2 weeks 
ago which came with Catalina.


Over the past week, I've been troubleshooting a massive jitter issue 
on Catalina, just between itself and my home router. For control, I 
have a Windows PC (tower-top) using a wireless adapter to connect to 
my home network. That has no jitter at all.


I have noticed as much as 300ms+ jitter on Catalina.

I then asked a few friends around the world to run tests for me on 
their own Catalina installations to their local router over wi-fi, 
and the results are the same. Jitter so high that what should be a 
1ms - 5ms latency can (for a short period) jump to 200ms+, 300ms+, 
400ms+.


On the off-chance that it is an issue with the new wireless chips on 
the later MacBook models, one of my friends tested the same on a 2013 
MacBook Pro running a beta version of Big Sur. Same story!


Another friend in South East Asia, testing on a 2018 13-inch MacBook 
Pro running Catalina, also had the same issue.


A Google search suggests that this is some known issue since Mojave, 
to do with Location Services, and some other apps, in a 
non-deterministic way:


https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router

For me, even after disabling all or some Location Services features, 
the problem remains.


Is anyone else seeing this on their Catalina Mac's while on wi-fi? If 
so, does anyone know what's going on here?


Ideally, this wouldn't matter if it was just a cosmetic issue - but I 
do actually see physical impact to performance of network access 
to/from the laptop, which has all the hallmarks of high jitter and/or 
packet loss.


An app like Zoom, which can display network performance data for a 
session in real-time, does indicate nominal packet loss for audio and 
video on this device, while other devices on the same WLAN are happy.


Thoughts?

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 16:08, J. Hellenthal wrote:


I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not 
diagnosing further because I could not find a reason for it fast enough and not 
sure if it really had an impact at the moment…. but could you try the following


sudo sysctl net.link.generic.system.hwcksum_tx=0
sudo sysctl net.link.generic.system.hwcksum_rx=0
sudo ifconfig en0 -rxcsum


Thanks, I'll have a sniff.


If you have some specific tests to run I would be willing to run them here on 
Big Sur with the same laptop but I have nothing now that runs Catalina


One of my mates found the same issue on Big Sur (beta) on a 2013 MacBook 
Pro.


Just a simple mtr test to your local home router's IP address should, 
over wi-fi, should show you the jitter.


Mark.

DE-CIX - Wednesday 11:00am - 1:30pm Eastern

[Matt Hoppes]

Does anyone here know any more about what crashed in the DE-CIX NYC 
exchange yesterday between approximately 11am and 1:30pm?


Between those times I'm told the Nokia switching fabric locked up and 
ultimately rebooted.

Join us Nov 13 for our first NANOG U Webinar!

[NANOG News]

College students and educators: Please join us on Friday, November 13 for
our first NANOG U Webinar! Learn from some of the brightest minds in our
community, and gain a competitive edge in Internet tech without leaving
home. Registration is free, and open to all. Learn More + Register Now


Not a student or educator? Help us spread the word, and pass this info
along to anyone you know who'd be interested in attending!

*Upcoming Webinars*

*2020*
*November 13 — Career Opportunities for Network Professionals*
11am - 1pm PST / 2pm - 4pm EST
Register Now 

*Speakers: *
Aaron Atac, Akamai Technologies
Hailey Dannenbring, Amazon
Kevin Epperson, Netflix
Selva Srinivasan, Microsoft

*Agenda: *
Panel of networking professionals with a Q to follow. Executive Director
Edward McNair will open with a discussion on NANOG.

*2021*
*January 22 — DNS Fundamentals*
11am - 1pm PST / 2pm - 4pm EST

*Speaker:*
Eddy Winstead, Internet Systems Consortium (ISC)

*February 26 — BGP Fundamentals*
11am - 1pm PST / 2pm - 4pm EST

*Speaker:*
Aaron Atac, Akamai Technologies

*Partner with us to empower and inspire.*
A NANOG Outreach partnership provides your organization a way to directly
support the communities who need access to our tools and resources most,
and the opportunity to inspire and connect with the next generation of
networking professionals. In turn, we’ll publicly recognize your
contributions as a partner in meaningful ways that resonate with the
communities we serve.

Interested in collaborating with us? Connect with Shawn Winstead
, NANOG's Business Development Specialist, to start a
conversation + learn more.

[NANOG-announce] Join us Nov 13 for our first NANOG U Webinar!

[NANOG News]

College students and educators: Please join us on Friday, November 13 for
our first NANOG U Webinar! Learn from some of the brightest minds in our
community, and gain a competitive edge in Internet tech without leaving
home. Registration is free, and open to all. Learn More + Register Now


Not a student or educator? Help us spread the word, and pass this info
along to anyone you know who'd be interested in attending!

*Upcoming Webinars*

*2020*
*November 13 — Career Opportunities for Network Professionals*
11am - 1pm PST / 2pm - 4pm EST
Register Now 

*Speakers: *
Aaron Atac, Akamai Technologies
Hailey Dannenbring, Amazon
Kevin Epperson, Netflix
Selva Srinivasan, Microsoft

*Agenda: *
Panel of networking professionals with a Q to follow. Executive Director
Edward McNair will open with a discussion on NANOG.

*2021*
*January 22 — DNS Fundamentals*
11am - 1pm PST / 2pm - 4pm EST

*Speaker:*
Eddy Winstead, Internet Systems Consortium (ISC)

*February 26 — BGP Fundamentals*
11am - 1pm PST / 2pm - 4pm EST

*Speaker:*
Aaron Atac, Akamai Technologies

*Partner with us to empower and inspire.*
A NANOG Outreach partnership provides your organization a way to directly
support the communities who need access to our tools and resources most,
and the opportunity to inspire and connect with the next generation of
networking professionals. In turn, we’ll publicly recognize your
contributions as a partner in meaningful ways that resonate with the
communities we serve.

Interested in collaborating with us? Connect with Shawn Winstead
, NANOG's Business Development Specialist, to start a
conversation + learn more.
___
NANOG-announce mailing list
NANOG-announce@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog-announce

Re: Apple Catalina Appears to Introduce Massive Jitter

[Niels Bakker]

* nanog@nanog.org (J. Hellenthal via NANOG) [Thu 29 Oct 2020, 15:10 CET]:
[disabling checksum offload]

Wireshark used to in Catalina rack up cksum errors a lot while these were all 
at their defaults.


This is completely expected behaviour for outgoing packets.


-- Niels.

Re: Apple Catalina Appears to Introduce Massive Jitter

[J. Hellenthal via NANOG]

Should also state here that net.inet.icmp.icmplim=0 and the command I have been 
testing from is: (ping -c 5000 -i 0.1 router)

--- router ping statistics ---
5000 packets transmitted, 5000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.118/4.060/172.031/6.841 ms


> On Oct 29, 2020, at 09:08, J. Hellenthal  wrote:
> 
> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. 
> Not diagnosing further because I could not find a reason for it fast enough 
> and not sure if it really had an impact at the moment…. but could you try the 
> following 
> 
> 
> sudo sysctl net.link.generic.system.hwcksum_tx=0
> sudo sysctl net.link.generic.system.hwcksum_rx=0
> sudo ifconfig en0 -rxcsum
> 
> 
> in reverse … to restore the settings 
> 
> sudo sysctl net.link.generic.system.hwcksum_tx=1
> sudo sysctl net.link.generic.system.hwcksum_rx=1
> sudo ifconfig en0 rxcsum
> 
> 
> If you have some specific tests to run I would be willing to run them here on 
> Big Sur with the same laptop but I have nothing now that runs Catalina
> 
> 
> Wireshark used to in Catalina rack up cksum errors a lot while these were all 
> at their defaults.
> 
> 
> 
>> On Oct 29, 2020, at 08:23, Mark Tinka  wrote:
>> 
>> 
>> 
>> On 10/29/20 15:04, Cory Sell wrote:
>> 
>>> Might be worth disabling each AP to see if there's one out there having an 
>>> issue playing nice with the MacBook. Also try different combinations of two 
>>> APs working together. It's possible the MacBook is flip flopping because 
>>> the power levels are fighting each other.
>> 
>> Tested all that, as well as dropping Tx power levels on each of the AP's to 
>> Low so that there isn't any power coming from any other AP (despite being 
>> quite far, already).
>> 
>> And to confirm, when the laptop locks into an AP, it doesn't try to join 
>> another one. When in range, power is very good (between -37dB and -52dB). 
>> When I walk away, that AP becomes too far (as bad as -80dB), but the next 
>> one close by is far better (same good values as before) and laptop connects 
>> and sticks to that.
>> 
>> Again, only impacts Catalina. No other Apple device, or the Windows PC that 
>> is on the same WLAN.
>> 
>> 
>>> Does the Mac have this issue at your local coffee shop or another 
>>> establishment with Wi-Fi? You can try to rule out the AirPort card in the 
>>> Mac itself.
>> 
>> Never tried, I generally work from home. If I'm out, it's faster to tether 
>> to my 4G service rather than any public wi-fi.
>> 
>> Mark.
> 
> 
> -- 
> 
> J. Hellenthal
> 
> The fact that there's a highway to Hell but only a stairway to Heaven says a 
> lot about anticipated traffic volume.
> 
> 
> 
> 
> 
> 


-- 

J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a 
lot about anticipated traffic volume.

Re: Apple Catalina Appears to Introduce Massive Jitter

[J. Hellenthal via NANOG]

I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not 
diagnosing further because I could not find a reason for it fast enough and not 
sure if it really had an impact at the moment…. but could you try the following 


sudo sysctl net.link.generic.system.hwcksum_tx=0
sudo sysctl net.link.generic.system.hwcksum_rx=0
sudo ifconfig en0 -rxcsum


in reverse … to restore the settings 

sudo sysctl net.link.generic.system.hwcksum_tx=1
sudo sysctl net.link.generic.system.hwcksum_rx=1
sudo ifconfig en0 rxcsum


If you have some specific tests to run I would be willing to run them here on 
Big Sur with the same laptop but I have nothing now that runs Catalina


Wireshark used to in Catalina rack up cksum errors a lot while these were all 
at their defaults.



> On Oct 29, 2020, at 08:23, Mark Tinka  wrote:
> 
> 
> 
> On 10/29/20 15:04, Cory Sell wrote:
> 
>> Might be worth disabling each AP to see if there's one out there having an 
>> issue playing nice with the MacBook. Also try different combinations of two 
>> APs working together. It's possible the MacBook is flip flopping because the 
>> power levels are fighting each other.
> 
> Tested all that, as well as dropping Tx power levels on each of the AP's to 
> Low so that there isn't any power coming from any other AP (despite being 
> quite far, already).
> 
> And to confirm, when the laptop locks into an AP, it doesn't try to join 
> another one. When in range, power is very good (between -37dB and -52dB). 
> When I walk away, that AP becomes too far (as bad as -80dB), but the next one 
> close by is far better (same good values as before) and laptop connects and 
> sticks to that.
> 
> Again, only impacts Catalina. No other Apple device, or the Windows PC that 
> is on the same WLAN.
> 
> 
>> Does the Mac have this issue at your local coffee shop or another 
>> establishment with Wi-Fi? You can try to rule out the AirPort card in the 
>> Mac itself.
> 
> Never tried, I generally work from home. If I'm out, it's faster to tether to 
> my 4G service rather than any public wi-fi.
> 
> Mark.


-- 

J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a 
lot about anticipated traffic volume.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 15:04, Cory Sell wrote:

Might be worth disabling each AP to see if there's one out there 
having an issue playing nice with the MacBook. Also try different 
combinations of two APs working together. It's possible the MacBook is 
flip flopping because the power levels are fighting each other.


Tested all that, as well as dropping Tx power levels on each of the AP's 
to Low so that there isn't any power coming from any other AP (despite 
being quite far, already).


And to confirm, when the laptop locks into an AP, it doesn't try to join 
another one. When in range, power is very good (between -37dB and 
-52dB). When I walk away, that AP becomes too far (as bad as -80dB), but 
the next one close by is far better (same good values as before) and 
laptop connects and sticks to that.


Again, only impacts Catalina. No other Apple device, or the Windows PC 
that is on the same WLAN.



Does the Mac have this issue at your local coffee shop or another 
establishment with Wi-Fi? You can try to rule out the AirPort card in 
the Mac itself.


Never tried, I generally work from home. If I'm out, it's faster to 
tether to my 4G service rather than any public wi-fi.


Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 14:40, Jared Mauch wrote:


I know there was a recent fix Apple did for devices talking to UBNT APs
for their handsets, perhaps there's a similar fix needed on your side?

I have all UBNT at home for wireless and periodically have some random
issues which I can't explain, but for the most part have things tuned to ensure
there's little to no interference.


I am running TP-Link AP's. Two of them are Google OnHub, which was built 
by TP-Link, and the 3rd one is the TP-Link Archer C6.


So they all support 802.11ac, which is where my device spends most of 
its time (5GHz).


No interference from my neighbors (separated by thick walls), and I am 
running separate channels for both frequencies per device.


Also, no other wireless device is suffering like this.



Do you see the same when hardwired?  I keep many of my devices hardwired
to avoid odd jitter issues.


No issues on the wire at all. Quite perfect.

Like you, I hard-wire all fixed devices (TV's, a/v receivers, satellite 
decoders, gaming consoles, energy meters, e.t.c.). The only devices on 
wireless are mobile devices, tablets, laptops and the Windows PC which 
is hooked up via wi-fi as well.




   I also saw some older versions of the Pulse Secure
VPN add the behavior you describe, including the more uptime the slower it would
get.


I use Viscosity as an SSL/VPN client. The issue is the same whether it 
is enabled or offline.


Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 14:37, Mike Hammett wrote:

Have you ruled out local wireless issues, such as a literal 
side-by-side test?


Yes - all other wi-fi devices don't exhibit this issue, including my 
wireless-connected PC. Only this MacBook running Catalina.


The problem exists at all wi-fi AP's around the house, all of which are 
easily 15m apart to cover the whole house.




Does the problem still exist when wired?


No problem on wire. As you can see below, my worst record latency value 
is 11.8ms only:


Marks-MacBook-Pro.local (172.16.0.239) 2020-10-29T14:48:12+0200
Keys:  Help   Display mode   Restart statistics   Order of fields quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.16.0.254 0.0%    50    1.0   2.6   0.7  11.8   2.1

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Jared Mauch]

On Thu, Oct 29, 2020 at 02:31:59PM +0200, Mark Tinka wrote:
> 
> 
> On 10/29/20 14:27, Matt Hoppes wrote:
> 
> > Is it actually jitter or is it potentially the wireless network card
> > going into sleep mode? I have seen that type of behavior on Apple
> > products when the cards go into low power mode although I can’t say I
> > have noticed that on my laptop.
> 
> Not, not sleep mode, because I am actively using the device to move data
> to/from the Internet.
> 
> I was actually struggling to upload some files to Youtube last weekend, and
> had to use another computer to do it as I couldn't figure out what was going
> on. That it is how bad the jitter is.
> 
> It seems to be a much bigger problem for the upload direction than the
> download, but it, inevitably creates a symmetrical performance problem.

I know there was a recent fix Apple did for devices talking to UBNT APs
for their handsets, perhaps there's a similar fix needed on your side?

I have all UBNT at home for wireless and periodically have some random
issues which I can't explain, but for the most part have things tuned to ensure
there's little to no interference.

Do you see the same when hardwired?  I keep many of my devices hardwired
to avoid odd jitter issues.  I also saw some older versions of the Pulse Secure
VPN add the behavior you describe, including the more uptime the slower it would
get.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mike Hammett]

Have you ruled out local wireless issues, such as a literal side-by-side test? 


Does the problem still exist when wired? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mark Tinka"  
To: "Matt Hoppes"  
Cc: "NANOG"  
Sent: Thursday, October 29, 2020 7:31:59 AM 
Subject: Re: Apple Catalina Appears to Introduce Massive Jitter 



On 10/29/20 14:27, Matt Hoppes wrote: 

> Is it actually jitter or is it potentially the wireless network card 
> going into sleep mode? I have seen that type of behavior on Apple 
> products when the cards go into low power mode although I can’t say I 
> have noticed that on my laptop. 

Not, not sleep mode, because I am actively using the device to move data 
to/from the Internet. 

I was actually struggling to upload some files to Youtube last weekend, 
and had to use another computer to do it as I couldn't figure out what 
was going on. That it is how bad the jitter is. 

It seems to be a much bigger problem for the upload direction than the 
download, but it, inevitably creates a symmetrical performance problem. 

Mark. 

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

On 10/29/20 14:27, Matt Hoppes wrote:

Is it actually jitter or is it potentially the wireless network card 
going into sleep mode? I have seen that type of behavior on Apple 
products when the cards go into low power mode although I can’t say I 
have noticed that on my laptop.


Not, not sleep mode, because I am actively using the device to move data 
to/from the Internet.


I was actually struggling to upload some files to Youtube last weekend, 
and had to use another computer to do it as I couldn't figure out what 
was going on. That it is how bad the jitter is.


It seems to be a much bigger problem for the upload direction than the 
download, but it, inevitably creates a symmetrical performance problem.


Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

Ah yes, an example of what I am seeing:


Marks-MacBook-Pro.local (172.16.0.239) 2020-10-29T14:28:27+0200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.16.0.254 0.8%   126    3.9  34.7   2.5 232.1  54.9

Mark.

On 10/29/20 14:07, Mark Tinka wrote:

Hi all.

I've been on High Sierra for several years now due to a limitation 
with an app that couldn't deal with Apple's latest rounds of system 
permissions since Mojave. Eventually, I gave up on waiting for them to 
fix it and upgraded my older Butterfly keyboard laptop to Catalina 4 
weeks ago.


At the same time, I picked up the new Magic keyboard laptop 2 weeks 
ago which came with Catalina.


Over the past week, I've been troubleshooting a massive jitter issue 
on Catalina, just between itself and my home router. For control, I 
have a Windows PC (tower-top) using a wireless adapter to connect to 
my home network. That has no jitter at all.


I have noticed as much as 300ms+ jitter on Catalina.

I then asked a few friends around the world to run tests for me on 
their own Catalina installations to their local router over wi-fi, and 
the results are the same. Jitter so high that what should be a 1ms - 
5ms latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.


On the off-chance that it is an issue with the new wireless chips on 
the later MacBook models, one of my friends tested the same on a 2013 
MacBook Pro running a beta version of Big Sur. Same story!


Another friend in South East Asia, testing on a 2018 13-inch MacBook 
Pro running Catalina, also had the same issue.


A Google search suggests that this is some known issue since Mojave, 
to do with Location Services, and some other apps, in a 
non-deterministic way:


https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router

For me, even after disabling all or some Location Services features, 
the problem remains.


Is anyone else seeing this on their Catalina Mac's while on wi-fi? If 
so, does anyone know what's going on here?


Ideally, this wouldn't matter if it was just a cosmetic issue - but I 
do actually see physical impact to performance of network access 
to/from the laptop, which has all the hallmarks of high jitter and/or 
packet loss.


An app like Zoom, which can display network performance data for a 
session in real-time, does indicate nominal packet loss for audio and 
video on this device, while other devices on the same WLAN are happy.


Thoughts?

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter

[Matt Hoppes]

Is it actually jitter or is it potentially the wireless network card going into 
sleep mode? I have seen that type of behavior on Apple products when the cards 
go into low power mode although I can’t say I have noticed that on my laptop.

> On Oct 29, 2020, at 8:11 AM, Mark Tinka  wrote:
> 
>  Hi all.
> 
> I've been on High Sierra for several years now due to a limitation with an 
> app that couldn't deal with Apple's latest rounds of system permissions since 
> Mojave. Eventually, I gave up on waiting for them to fix it and upgraded my 
> older Butterfly keyboard laptop to Catalina 4 weeks ago.
> 
> At the same time, I picked up the new Magic keyboard laptop 2 weeks ago which 
> came with Catalina.
> 
> Over the past week, I've been troubleshooting a massive jitter issue on 
> Catalina, just between itself and my home router. For control, I have a 
> Windows PC (tower-top) using a wireless adapter to connect to my home 
> network. That has no jitter at all.
> 
> I have noticed as much as 300ms+ jitter on Catalina.
> 
> I then asked a few friends around the world to run tests for me on their own 
> Catalina installations to their local router over wi-fi, and the results are 
> the same. Jitter so high that what should be a 1ms - 5ms latency can (for a 
> short period) jump to 200ms+, 300ms+, 400ms+.
> 
> On the off-chance that it is an issue with the new wireless chips on the 
> later MacBook models, one of my friends tested the same on a 2013 MacBook Pro 
> running a beta version of Big Sur. Same story!
> 
> Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro 
> running Catalina, also had the same issue.
> 
> A Google search suggests that this is some known issue since Mojave, to do 
> with Location Services, and some other apps, in a non-deterministic way:
> 
> 
> https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router
> 
> For me, even after disabling all or some Location Services features, the 
> problem remains.
> 
> Is anyone else seeing this on their Catalina Mac's while on wi-fi? If so, 
> does anyone know what's going on here?
> 
> Ideally, this wouldn't matter if it was just a cosmetic issue - but I do 
> actually see physical impact to performance of network access to/from the 
> laptop, which has all the hallmarks of high jitter and/or packet loss. 
> 
> An app like Zoom, which can display network performance data for a session in 
> real-time, does indicate nominal packet loss for audio and video on this 
> device, while other devices on the same WLAN are happy.
> 
> Thoughts?
> 
> Mark.

Apple Catalina Appears to Introduce Massive Jitter

[Mark Tinka]

Hi all.

I've been on High Sierra for several years now due to a limitation with 
an app that couldn't deal with Apple's latest rounds of system 
permissions since Mojave. Eventually, I gave up on waiting for them to 
fix it and upgraded my older Butterfly keyboard laptop to Catalina 4 
weeks ago.


At the same time, I picked up the new Magic keyboard laptop 2 weeks ago 
which came with Catalina.


Over the past week, I've been troubleshooting a massive jitter issue on 
Catalina, just between itself and my home router. For control, I have a 
Windows PC (tower-top) using a wireless adapter to connect to my home 
network. That has no jitter at all.


I have noticed as much as 300ms+ jitter on Catalina.

I then asked a few friends around the world to run tests for me on their 
own Catalina installations to their local router over wi-fi, and the 
results are the same. Jitter so high that what should be a 1ms - 5ms 
latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.


On the off-chance that it is an issue with the new wireless chips on the 
later MacBook models, one of my friends tested the same on a 2013 
MacBook Pro running a beta version of Big Sur. Same story!


Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro 
running Catalina, also had the same issue.


A Google search suggests that this is some known issue since Mojave, to 
do with Location Services, and some other apps, in a non-deterministic way:


https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router

For me, even after disabling all or some Location Services features, the 
problem remains.


Is anyone else seeing this on their Catalina Mac's while on wi-fi? If 
so, does anyone know what's going on here?


Ideally, this wouldn't matter if it was just a cosmetic issue - but I do 
actually see physical impact to performance of network access to/from 
the laptop, which has all the hallmarks of high jitter and/or packet loss.


An app like Zoom, which can display network performance data for a 
session in real-time, does indicate nominal packet loss for audio and 
video on this device, while other devices on the same WLAN are happy.


Thoughts?

Mark.

Re: Asus wifi AP re-writing DNS packets

[Alarig Le Lay]

On Thu 29 Oct 2020 02:10:25 GMT, Anurag Bhatia wrote:
> I tried deleting the rule and it drops the traffic completely. So DNS
> resolution stops working and I am unsure why. It's not like default drop or
> anything. I can edit the rule and whatever active port 53 related rule is
> there works. But I want case of no such rule at all. :-)

Did you try to add
-t nat -A POSTROUTING -p tcp -m tcp --dport 53 -j ACCEPT
-t nat -A POSTROUTING -p udp -m udp --dport 53 -j ACCEPT

after the deletion?

-- 
Alarig