Re: [Cake] Ubiquity (Unifi ) Smart Queues

[dave seddon via Cake]

G'day,

Just a small update on the Unifi security gateway stuff.  They have a new
range of devices which are a lot more powerful.
(
https://store.ui.com/us/en/collections/cloud-gateway-ultra/products/ucg-ultra
)

The good news is that the limits set in the GUI now match exactly the
"rate" set in the qcdisc.

root@UCG-Ultra:~# *tc -p -s -d qdisc show dev eth4*
qdisc htb 1: root refcnt 5 r2q 10 default 0x2 direct_packets_stat 0 ver
3.17 direct_qlen 1000
 Sent 13112672757 bytes 41407610 pkt (dropped 2863, overlimits 12123381
requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 2: parent 1:2 limit 2000p flows 1024 quantum 300 target 5ms
interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 13112672757 bytes 41407610 pkt (dropped 2863, overlimits 0 requeues
0)
 backlog 0b 0p requeues 0
  maxpacket 27888 drop_overlimit 0 new_flow_count 9175282 ecn_mark 0
  new_flows_len 1 old_flows_len 3
qdisc ingress : parent :fff1 
 Sent 104038056896 bytes 143646981 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

root@UCG-Ultra:/etc/systemd# *tc -d class show dev eth4*
class htb 1:1 root rate 35Mbit ceil 35Mbit linklayer ethernet burst 1491b/1
mpu 0b cburst 1491b/1 mpu 0b level 7
class htb 1:2 parent 1:1 leaf 2: prio 7 quantum 1514 rate 64bit ceil 35Mbit
linklayer ethernet burst 1500b/1 mpu 0b cburst 1491b/1 mpu 0b level 0
class fq_codel 2:1bf parent 2:
class fq_codel 2:274 parent 2:
class fq_codel 2:296 parent 2:
class fq_codel 2:2ca parent 2:
class fq_codel 2:34a parent 2:
class fq_codel 2:364 parent 2:

root@UCG-Ultra:~# *tc -p -s -d qdisc show dev ifbeth4*
qdisc htb 1: root refcnt 2 r2q 10 default 0x2 direct_packets_stat 0 ver
3.17 direct_qlen 1000
 Sent 108770017013 bytes 143572868 pkt (dropped 24028, overlimits 43487579
requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 2: parent 1:2 limit 2000p flows 1024 quantum 1514 target 5ms
interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 108770017013 bytes 143572868 pkt (dropped 24028, overlimits 0
requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 69876 drop_overlimit 10448 new_flow_count 14414347 ecn_mark 0
drop_overmemory 10448
  new_flows_len 1 old_flows_len 2

root@UCG-Ultra:/etc/systemd# *tc -d class show dev ifbeth4*
class htb 1:1 root rate 800Mbit ceil 800Mbit linklayer ethernet burst
1400b/1 mpu 0b cburst 1400b/1 mpu 0b level 7
class htb 1:2 parent 1:1 leaf 2: prio 7 quantum 1514 rate 64bit ceil
800Mbit linklayer ethernet burst 1500b/1 mpu 0b cburst 1400b/1 mpu 0b level
0
class fq_codel 2:111 parent 2:
class fq_codel 2:3cc parent 2:

So 35Mb/s and 800Mb/s match what is configured in the GUI.

[image: image.png]

The bad news is still no cake.

The bottleneck in my house is now the air interfaces.   I'll run some flent
tests soon.

Thanks,
Dave Seddon


Other device details

root@UCG-Ultra:~# uname -a
Linux UCG-Ultra 5.4.213-ui-ipq5322 #5.4.213 SMP PREEMPT Fri Jan 26 01:53:55
CST 2024 aarch64 GNU/Linux

root@UCG-Ultra:~# cat /proc/cpuinfo
processor : 0
BogoMIPS : 48.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x51
CPU architecture: 8
CPU variant : 0xa
CPU part : 0x801
CPU revision : 4

processor : 1
BogoMIPS : 48.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x51
CPU architecture: 8
CPU variant : 0xa
CPU part : 0x801
CPU revision : 4

processor : 2
BogoMIPS : 48.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x51
CPU architecture: 8
CPU variant : 0xa
CPU part : 0x801
CPU revision : 4

processor : 3
BogoMIPS : 48.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x51
CPU architecture: 8
CPU variant : 0xa
CPU part : 0x801
CPU revision : 4


root@UCG-Ultra:~# cat /proc/interrupts
   CPU0   CPU1   CPU2   CPU3
  4:   49385470   71684295   74561605   77496134 GIC-0  20 Level
arch_timer
  6:  0  0  0  0 GIC-0  39 Level
arch_mem_timer
  8:  0  0  0  0 GIC-0 195 Level
edma_txcmpl_4
  9:  0  0  0  0 GIC-0 196 Level
edma_txcmpl_5
 10:  0  0  0  0 GIC-0 197 Level
edma_txcmpl_6
 11:  0  0  0  0 GIC-0 198 Level
edma_txcmpl_7
 12:1301701  0  0  0 GIC-0 199 Level
edma_txcmpl_8
 13:   16537922  0  0  0 GIC-0 200 Level
edma_txcmpl_9
 14:   16902391  0  0  0 GIC-0 201 Level
edma_txcmpl_10
 15:   19093638  0  0  0 GIC-0 202 Level
edma_txcmpl_11
 16: 218358  0  0  0 GIC-0 203 Level
edma_txcmpl_12
 17:   14172534  0  0  0 GIC-0 204 Level
edma_txcmpl_13
 18:   12228644  0  0  0 GIC-0 205 Level
edma_txcmpl_14
 19:   14848643  0  0  0 GIC-0 206 Level
edma_txcmpl_15
 20:  

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Nils Andreas Svee via Cake]

Well my NIC has 4 queues as far as I can tell, so it could likely work, but as 
you say it’s like killing a mosquito with a gatling gun.

Those graphs are sweet though, and it’s been in my backlog for awhile to do 
something with Grafana to get something similar, like this one from a few years 
ago you’ve seen too: https://forum.openwrt.org/t/sqm-reporting/59960/96

Best Regards,
Nils Andreas Svee

> On Jan 9, 2024, at 18:17, Dave Taht  wrote:
> 
> Principal limitation for libreqos on a small box is has to have
> multiple hardware queues and support eBPF.
> 
> Seriously folks, running libreqos at home is *serious overkill*,
> although I have to admit the traffic graphs are mesmerizing!!! One of
> our ISPs has been setting them to music:
> https://www.youtube.com/@trendaltoews7143
> 
> Herbert has been working on adding all sorts of other analytics to it also.
> 
> On Tue, Jan 9, 2024 at 12:07 PM dave seddon  wrote:
>> 
>> Nils - I guess you could run LibreQoS on N100?
>> 
>> On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake 
>>  wrote:
>>> 
>>> On Jan 9, 2024, at 17:05, Dave Taht  wrote:
>>> 
>>> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
>>>  wrote:
>>> 
>>> Though frankly, I don’t plan on updating the sch_cake and tc binaries when 
>>> new firmwares are released anymore, as they don’t publish the GPL archives 
>>> on their webpage after the redesign, and they don’t respond to requests for 
>>> them either by the looks of the forums. So if it breaks there’s not much I 
>>> can do anymore.
>>> 
>>> 
>>> This irks me enormously. It is the direct outcome of the cambium
>>> elevate lawsuit, where both companies lost, the ISPs lost, open source
>>> practices long established about publishing sources, lost, and the
>>> lawyers went on to other nasty things leaving this trail of awful
>>> precedents  in their wake.
>>> 
>>> https://www.mtin.net/blog/ubnt-vs-cambium/
>>> 
>>> Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s 
>>> software on their hardware?
>>> That is crazy, just evil corporate lawyers doing their thing I guess.
>>> 
>>> I do not know what to do about it. It also irks me that as a
>>> contributor to "smart queues" they are not maintaining it well.
>>> 
>>> It leaves something to be desired yes, and I would’ve hoped to see CAKE 
>>> included too of course,
>>> but even WireGuard is only available in the latest release candidates with 
>>> the redesigned web UI, so I’m not holding my breath.
>>> 
>>> I still have an EdgeRouter 4 that serves the family farm and one of the 
>>> 8-port switches under my desk, if only because I don’t wanna spend money on 
>>> replacing them, and they do serve their purpose.
>>> 
>>> I’ve since moved though, and now live in an area that has FTTH, so I needed 
>>> something beefier to handle CAKE on a 750/750 subscription, because 
>>> obviously there’s still bloat even on that ;)
>>> 
>>> One of those Chinese boxes with a N100 in it and OpenWrt on top works 
>>> wonders :)
>>> 
>>> Best Regards,
>>> Nils Andreas Svee
>>> ___
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>> 
>> 
>> 
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
> 
> 
> 
> -- 
> 40 years of net history, a couple songs:
> https://www.youtube.com/watch?v=D9RGX6QFm5E
> Dave Täht CSO, LibreQos

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Nils Andreas Svee via Cake]

I probably could, but it seems *a bit* more complex than I need more my little 
home network? ;)

Best Regards,
Nils Andreas Svee

> On Jan 9, 2024, at 18:07, dave seddon  wrote:
> 
> Nils - I guess you could run LibreQoS on N100?
> 
> On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake 
> mailto:cake@lists.bufferbloat.net>> wrote:
>>> On Jan 9, 2024, at 17:05, Dave Taht >> > wrote:
>>> 
>>> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
>>> mailto:cake@lists.bufferbloat.net>> wrote:
>>> 
 Though frankly, I don’t plan on updating the sch_cake and tc binaries when 
 new firmwares are released anymore, as they don’t publish the GPL archives 
 on their webpage after the redesign, and they don’t respond to requests 
 for them either by the looks of the forums. So if it breaks there’s not 
 much I can do anymore.
>>> 
>>> This irks me enormously. It is the direct outcome of the cambium
>>> elevate lawsuit, where both companies lost, the ISPs lost, open source
>>> practices long established about publishing sources, lost, and the
>>> lawyers went on to other nasty things leaving this trail of awful
>>> precedents  in their wake.
>>> https://www.mtin.net/blog/ubnt-vs-cambium/
>> Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s 
>> software on their hardware?
>> That is crazy, just evil corporate lawyers doing their thing I guess.
>> 
>>> I do not know what to do about it. It also irks me that as a
>>> contributor to "smart queues" they are not maintaining it well.
>> It leaves something to be desired yes, and I would’ve hoped to see CAKE 
>> included too of course,
>> but even WireGuard is only available in the latest release candidates with 
>> the redesigned web UI, so I’m not holding my breath.
>> 
>> I still have an EdgeRouter 4 that serves the family farm and one of the 
>> 8-port switches under my desk, if only because I don’t wanna spend money on 
>> replacing them, and they do serve their purpose.
>> 
>> I’ve since moved though, and now live in an area that has FTTH, so I needed 
>> something beefier to handle CAKE on a 750/750 subscription, because 
>> obviously there’s still bloat even on that ;)
>> 
>> One of those Chinese boxes with a N100 in it and OpenWrt on top works 
>> wonders :)
>> 
>> Best Regards,
>> Nils Andreas Svee
>> ___
>> Cake mailing list
>> Cake@lists.bufferbloat.net 
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> --
> Regards,
> Dave Seddon
> +1 415 857 5102

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Dave Taht via Cake]

Principal limitation for libreqos on a small box is has to have
multiple hardware queues and support eBPF.

Seriously folks, running libreqos at home is *serious overkill*,
although I have to admit the traffic graphs are mesmerizing!!! One of
our ISPs has been setting them to music:
https://www.youtube.com/@trendaltoews7143

Herbert has been working on adding all sorts of other analytics to it also.

On Tue, Jan 9, 2024 at 12:07 PM dave seddon  wrote:
>
> Nils - I guess you could run LibreQoS on N100?
>
> On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake 
>  wrote:
>>
>> On Jan 9, 2024, at 17:05, Dave Taht  wrote:
>>
>> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
>>  wrote:
>>
>> Though frankly, I don’t plan on updating the sch_cake and tc binaries when 
>> new firmwares are released anymore, as they don’t publish the GPL archives 
>> on their webpage after the redesign, and they don’t respond to requests for 
>> them either by the looks of the forums. So if it breaks there’s not much I 
>> can do anymore.
>>
>>
>> This irks me enormously. It is the direct outcome of the cambium
>> elevate lawsuit, where both companies lost, the ISPs lost, open source
>> practices long established about publishing sources, lost, and the
>> lawyers went on to other nasty things leaving this trail of awful
>> precedents  in their wake.
>>
>> https://www.mtin.net/blog/ubnt-vs-cambium/
>>
>> Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s 
>> software on their hardware?
>> That is crazy, just evil corporate lawyers doing their thing I guess.
>>
>> I do not know what to do about it. It also irks me that as a
>> contributor to "smart queues" they are not maintaining it well.
>>
>> It leaves something to be desired yes, and I would’ve hoped to see CAKE 
>> included too of course,
>> but even WireGuard is only available in the latest release candidates with 
>> the redesigned web UI, so I’m not holding my breath.
>>
>> I still have an EdgeRouter 4 that serves the family farm and one of the 
>> 8-port switches under my desk, if only because I don’t wanna spend money on 
>> replacing them, and they do serve their purpose.
>>
>> I’ve since moved though, and now live in an area that has FTTH, so I needed 
>> something beefier to handle CAKE on a 750/750 subscription, because 
>> obviously there’s still bloat even on that ;)
>>
>> One of those Chinese boxes with a N100 in it and OpenWrt on top works 
>> wonders :)
>>
>> Best Regards,
>> Nils Andreas Svee
>> ___
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[dave seddon via Cake]

Nils - I guess you could run LibreQoS on N100?

On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake <
cake@lists.bufferbloat.net> wrote:

> On Jan 9, 2024, at 17:05, Dave Taht  wrote:
>
> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
>  wrote:
>
> Though frankly, I don’t plan on updating the sch_cake and tc binaries when
> new firmwares are released anymore, as they don’t publish the GPL archives
> on their webpage after the redesign, and they don’t respond to requests for
> them either by the looks of the forums. So if it breaks there’s not much I
> can do anymore.
>
>
> This irks me enormously. It is the direct outcome of the cambium
> elevate lawsuit, where both companies lost, the ISPs lost, open source
> practices long established about publishing sources, lost, and the
> lawyers went on to other nasty things leaving this trail of awful
> precedents  in their wake.
>
> https://www.mtin.net/blog/ubnt-vs-cambium/
>
> Wow, hadn’t read about that. They even sued an ISP just for using
> Cambium’s software on their hardware?
> That is crazy, just evil corporate lawyers doing their thing I guess.
>
> I do not know what to do about it. It also irks me that as a
> contributor to "smart queues" they are not maintaining it well.
>
> It leaves something to be desired yes, and I would’ve hoped to see CAKE
> included too of course,
> but even WireGuard is only available in the latest release candidates with
> the redesigned web UI, so I’m not holding my breath.
>
> I still have an EdgeRouter 4 that serves the family farm and one of the
> 8-port switches under my desk, if only because I don’t wanna spend money on
> replacing them, and they do serve their purpose.
>
> I’ve since moved though, and now live in an area that has FTTH, so I
> needed something beefier to handle CAKE on a 750/750 subscription, because
> obviously there’s still bloat even on that ;)
>
> One of those Chinese boxes with a N100 in it and OpenWrt on top works
> wonders :)
>
> Best Regards,
> Nils Andreas Svee
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>


-- 
Regards,
Dave Seddon
+1 415 857 5102
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Nils Andreas Svee via Cake]

> On Jan 9, 2024, at 17:05, Dave Taht  wrote:
> 
> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
> mailto:cake@lists.bufferbloat.net>> wrote:
> 
>> Though frankly, I don’t plan on updating the sch_cake and tc binaries when 
>> new firmwares are released anymore, as they don’t publish the GPL archives 
>> on their webpage after the redesign, and they don’t respond to requests for 
>> them either by the looks of the forums. So if it breaks there’s not much I 
>> can do anymore.
> 
> This irks me enormously. It is the direct outcome of the cambium
> elevate lawsuit, where both companies lost, the ISPs lost, open source
> practices long established about publishing sources, lost, and the
> lawyers went on to other nasty things leaving this trail of awful
> precedents  in their wake.
> https://www.mtin.net/blog/ubnt-vs-cambium/
Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s 
software on their hardware?
That is crazy, just evil corporate lawyers doing their thing I guess.

> I do not know what to do about it. It also irks me that as a
> contributor to "smart queues" they are not maintaining it well.
It leaves something to be desired yes, and I would’ve hoped to see CAKE 
included too of course,
but even WireGuard is only available in the latest release candidates with the 
redesigned web UI, so I’m not holding my breath.

I still have an EdgeRouter 4 that serves the family farm and one of the 8-port 
switches under my desk, if only because I don’t wanna spend money on replacing 
them, and they do serve their purpose.

I’ve since moved though, and now live in an area that has FTTH, so I needed 
something beefier to handle CAKE on a 750/750 subscription, because obviously 
there’s still bloat even on that ;)

One of those Chinese boxes with a N100 in it and OpenWrt on top works wonders :)

Best Regards,
Nils Andreas Svee___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[dave seddon via Cake]

Thanks to everyone for the comments.

Wow Dave.  Thanks for the links to the lawsuit.  Fascinating.  I didn't
know about this.

Since the original email, I actually downgraded from Spectrum cable 300
Mb/s to ? maybe it's 200 now ?, anyway it's $20 a month cheaper.  And
decreased the "smart queue" limits to 80/10 Mb/s.

Video streaming multiple TVs at once is all working well.  No complaints
from the family. Win!

Stats are looking pretty good after ~8 days uptime

root@USG-Pro-4:/home/daveseddon# tc -p -s -d qdisc show
qdisc fq_codel 8001: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms ecn
 Sent 161758598841 bytes 147108740 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 299 ecn_mark 0
  new_flows_len 0 old_flows_len 4
qdisc htb 1: dev eth2 root refcnt 2 r2q 10 default 10 direct_packets_stat 0
ver 3.17
 Sent 32491619852 bytes 93191686 pkt (dropped 0, overlimits 3643
requeues 0)< OVER ( 3643 / 32491619852
~= 0.00112)
 backlog 0b 0p requeues 0
qdisc fq_codel 100: dev eth2 parent 1:10 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms ecn
 Sent 32491619852 bytes 93191686 pkt (dropped 37156, overlimits 0 requeues
0)<--- DROPS ( 37156 / 32491619852
~= 0.011 )
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 15549718 ecn_mark
1572   <--- some ECN
  new_flows_len 1 old_flows_len 5
qdisc ingress : dev eth2 parent :fff1 
 Sent 169147707173 bytes 346339031 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc htb 1: dev ifb_eth2 root refcnt 2 r2q 10 default 10
direct_packets_stat 0 ver 3.17
 Sent 173801196835 bytes 346221051 pkt (dropped 0, overlimits 192974926
requeues 0)   <--- OVER ( 192974926 / 173801196835
~= 0.0011 )
 backlog 0b 0p requeues 0
qdisc fq_codel 100: dev ifb_eth2 parent 1:10 limit 10240p flows 1024
quantum 1514 target 5.0ms interval 100.0ms ecn
 Sent 173801196835 bytes 346221051 pkt (dropped 140361, overlimits 0
requeues 0)   <--- DROPS ( 37156 / 32491619852
~= 0.0080 )
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 30970803 ecn_mark
1721  <--- some ECN
  new_flows_len 1 old_flows_len 3

root@USG-Pro-4:/home/daveseddon# tc -d class show dev eth2
class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil
9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b
level 0
class fq_codel 100:98 parent 100:
class fq_codel 100:c7 parent 100:
class fq_codel 100:180 parent 100:
class fq_codel 100:238 parent 100:
class fq_codel 100:305 parent 100:

root@USG-Pro-4:/home/daveseddon# tc -d class show dev ifb_eth2
class htb 1:10 root leaf 100: prio 0 quantum 20 rate 76000Kbit ceil
76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead
0b level 0
class fq_codel 100:247 parent 100:
class fq_codel 100:2c8 parent 100:

root@USG-Pro-4:/home/daveseddon# uptime
 08:18:21 up 8 days, 19:02,  1 user,  load average: 0.00, 0.01, 0.05

root@USG-Pro-4:/home/daveseddon# netstat -ia
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR
Flg
eth0   1500 0  101353919  0  0 0  157787555  0  0
   0 BMRU
eth1   1500 0663477  0  0 0   1090665  0  0
 0 BMRU
eth2   1500 0  377699134  0  0 0  98049876  0  0
   0 BMRU
eth3   1500 0 0  0  0 0 0  0  0
 0 BM
eth0.201500 0   3437713  0  0 0   2260022  0  0
 0 BMRU
eth0.401500 0 12524  0  0 0   1012668  0  0
 0 BMRU
ifb_eth2   1500 0  34608  0  22391 0  346310917  0  0
   0 BORU   <--- i'm still curious about these drops, but it's a tiny
amount. 22391 / 34608 ~= 0.64
imq0  16000 0 0  0  0 0 0  0  0
 0 ORU
lo65536 0 38330  0  0 0 38330  0  0
 0 LRU
loop0  1500 0 0  0  0 0 0  0  0
 0 BM
loop1  1500 0 0  0  0 0 0  0  0
 0 BM
loop2  1500 0 0  0  0 0 0  0  0
 0 BM
loop3  1500 0 0  0  0 0 0  0  0
 0 BM
npi0   1500 0 0  0  0 0 0  0  0
 0 BM
npi1   1500 0 0  0  0 0 0  0  0
 0 BM
npi2   1500 0 0  0  0 0 0  0  0
 0 BM
npi3   1500 0 0  0  0 0 0  0  0
 0 BM





On Tue, Jan 9, 

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Dave Taht via Cake]

On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
 wrote:

> Though frankly, I don’t plan on updating the sch_cake and tc binaries when 
> new firmwares are released anymore, as they don’t publish the GPL archives on 
> their webpage after the redesign, and they don’t respond to requests for them 
> either by the looks of the forums. So if it breaks there’s not much I can do 
> anymore.

This irks me enormously. It is the direct outcome of the cambium
elevate lawsuit, where both companies lost, the ISPs lost, open source
practices long established about publishing sources, lost, and the
lawyers went on to other nasty things leaving this trail of awful
precedents  in their wake.

https://www.mtin.net/blog/ubnt-vs-cambium/

I do not know what to do about it. It also irks me that as a
contributor to "smart queues" they are not maintaining it well.

>
> Best Regards,
> Nils Andreas Svee
>
> On Jan 3, 2024, at 14:44, Pete Heist via Cake  
> wrote:
>
> On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote:
>
> I thought people might be interested to see what Ubiquity/Unifi is
> doing with "Smart Queues" on their devices.  The documentation on
> their website is not very informative.
> 
> "Smart Queue" Implementation
>
> Looks like they only apply tc qdiscs to the Eth2, and sadly this is
> NOT cake, but fq_codel.
>
> And cake isn't available :(
>
> root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt
> 20ms
> Unknown qdisc "cake", hence option "bandwidth" is unparsable
>
>
> Hi Dave, there's a community contributed version of Cake for EdgeRouter
> devices that I've been using for years on production ER-X's:
>
> https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2
>
> I don't think that works for UniFi/USG devices, however, and one should
> note the disclaimer and be careful when installing it. Also, it must be
> re-installed after every upgrade.
>
> Cheers,
> Pete
>
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Dave Taht via Cake]

It is so nice to see this list come to life again.
I just wanted to point out that the inbound drop rate was merely .03%
in the first email.

Elsewhere we keep hearing of 1-2% drop rates being common, and I just
aint seeing it in any of the larger scale data I have been getting
from the libreqos deployment. Maybe that´s how fifos collapse in slow
start a lot more than we have ever observed. Maybe it is retransmits
going wild at 250ms worth of buffering.

scale=10
10688/29185742
.0003662062

I still have to sit back and admire you all for this magnificent
achievement of cake. I am not really the gloating sort, but seeing
juniper go up for sale to hpe today, at a firesale price, is oddly
satisfying.

https://blog.cerowrt.org/post/juniper/



On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake
 wrote:
>
> You’re unlikely to do any real harm though, but the warning is there cause 
> you can potentially soft brick your router using it. I’ve run into that 
> myself if I remember correctly, where after a firmware upgrade the kernel had 
> slightly changed, so loading the sch_cake module caused it to panic. And I 
> had it start through /config/scripts/post-config.d of course, so it would 
> happen on every restart.
>
> Nothing a factory reset won’t solve, but annoying when if you’re messing 
> about remotely :)
>
> As for USG, I think I used to have some binaries for those too. I do still 
> have some old kernel sources for them laying around in a repo.
> It’s been awhile, but I probably stopped building for those as it wasn’t as 
> straightforward to keep up with the versions of the firmware.
>
> Though frankly, I don’t plan on updating the sch_cake and tc binaries when 
> new firmwares are released anymore, as they don’t publish the GPL archives on 
> their webpage after the redesign, and they don’t respond to requests for them 
> either by the looks of the forums. So if it breaks there’s not much I can do 
> anymore.
>
> Best Regards,
> Nils Andreas Svee
>
> On Jan 3, 2024, at 14:44, Pete Heist via Cake  
> wrote:
>
> On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote:
>
> I thought people might be interested to see what Ubiquity/Unifi is
> doing with "Smart Queues" on their devices.  The documentation on
> their website is not very informative.
> 
> "Smart Queue" Implementation
>
> Looks like they only apply tc qdiscs to the Eth2, and sadly this is
> NOT cake, but fq_codel.
>
> And cake isn't available :(
>
> root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt
> 20ms
> Unknown qdisc "cake", hence option "bandwidth" is unparsable
>
>
> Hi Dave, there's a community contributed version of Cake for EdgeRouter
> devices that I've been using for years on production ER-X's:
>
> https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2
>
> I don't think that works for UniFi/USG devices, however, and one should
> note the disclaimer and be careful when installing it. Also, it must be
> re-installed after every upgrade.
>
> Cheers,
> Pete
>
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Nils Andreas Svee via Cake]

You’re unlikely to do any real harm though, but the warning is there cause you 
can potentially soft brick your router using it. I’ve run into that myself if I 
remember correctly, where after a firmware upgrade the kernel had slightly 
changed, so loading the sch_cake module caused it to panic. And I had it start 
through /config/scripts/post-config.d of course, so it would happen on every 
restart.

Nothing a factory reset won’t solve, but annoying when if you’re messing about 
remotely :)

As for USG, I think I used to have some binaries for those too. I do still have 
some old kernel sources for them laying around in a repo.
It’s been awhile, but I probably stopped building for those as it wasn’t as 
straightforward to keep up with the versions of the firmware.

Though frankly, I don’t plan on updating the sch_cake and tc binaries when new 
firmwares are released anymore, as they don’t publish the GPL archives on their 
webpage after the redesign, and they don’t respond to requests for them either 
by the looks of the forums. So if it breaks there’s not much I can do anymore.

Best Regards,
Nils Andreas Svee

> On Jan 3, 2024, at 14:44, Pete Heist via Cake  
> wrote:
> 
> On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote:
>> I thought people might be interested to see what Ubiquity/Unifi is
>> doing with "Smart Queues" on their devices.  The documentation on
>> their website is not very informative.
>> 
>> "Smart Queue" Implementation
>> 
>> Looks like they only apply tc qdiscs to the Eth2, and sadly this is
>> NOT cake, but fq_codel.
>> 
>> And cake isn't available :(
>> 
>> root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt
>> 20ms
>> Unknown qdisc "cake", hence option "bandwidth" is unparsable
> 
> Hi Dave, there's a community contributed version of Cake for EdgeRouter
> devices that I've been using for years on production ER-X's:
> 
> https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2
> 
> I don't think that works for UniFi/USG devices, however, and one should
> note the disclaimer and be careful when installing it. Also, it must be
> re-installed after every upgrade.
> 
> Cheers,
> Pete
> 
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Pete Heist via Cake]

On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote:
> I thought people might be interested to see what Ubiquity/Unifi is
> doing with "Smart Queues" on their devices.  The documentation on
> their website is not very informative.
> 
> "Smart Queue" Implementation
> 
> Looks like they only apply tc qdiscs to the Eth2, and sadly this is
> NOT cake, but fq_codel.
> 
> And cake isn't available :(
> 
> root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt
> 20ms
> Unknown qdisc "cake", hence option "bandwidth" is unparsable

Hi Dave, there's a community contributed version of Cake for EdgeRouter
devices that I've been using for years on production ER-X's:

https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2

I don't think that works for UniFi/USG devices, however, and one should
note the disclaimer and be careful when installing it. Also, it must be
re-installed after every upgrade.

Cheers,
Pete

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Jonathan Morton via Cake]

> On 2 Jan, 2024, at 8:59 pm, dave seddon via Cake  
> wrote:
> 
>   • I'm not really sure what "overlimits" means or what that does, and 
> tried looking this up, but I guess the kernel source is likely the "best" 
> documentation for this.  Maybe this means it's dropping?  Or is it ECN?

Overlimit just means the shaper in HTB is restricting the flow, thus doing 
something useful.  Drop means the AQM is working to provide congestion 
information to a Not-ECT flow.  Those numbers look normal (about 0.2% drop 
rate, and most packets hitting "overlimit" when the link is saturated).

Using fq_codel's default parameters is not a bad thing at all.  I'd much rather 
they did that, thereby using numbers that are tuned for general Internet 
conditions, than try to change that tuning in ignorance of the end-user's 
actual network environment.  Most end-users have their WAN port facing "general 
Internet conditions" anyway.

> Apparently rather than applying the tc qdsic on the outbound path on the LAN 
> side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2.

That's the correct place to do it, so that the qdisc applies specifically to 
traffic coming from the WAN.  If you apply it to the LAN egress, it tends to 
affect traffic coming through the router from the WiFi or its internal servers, 
which is not desirable.

These are all questions we've considered at length in the process of developing 
and deploying Cake and other SQM solutions.

 - Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Sebastian Moeller via Cake]

Hi Dave,


> On Jan 2, 2024, at 22:15, dave seddon  wrote:
> 
> Thanks Sebastian!
> 
> Now I see the rates!!
> 
> I actually reduced the rates to ensure this device is the bottleneck 80/10 
> Mb/s
> 
> 
> 
> root@USG-Pro-4:~# tc -d class show dev eth2
> class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil 
> 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b 
> level 0 
> class fq_codel 100:12c parent 100: 
> class fq_codel 100:213 parent 100: 
> class fq_codel 100:22e parent 100: 
> 
> root@USG-Pro-4:~# tc -d class show dev ifb_eth2
> class htb 1:10 root leaf 100: prio 0 quantum 20 rate 76000Kbit ceil 
> 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b 
> level 0 
> class fq_codel 100:2c8 parent 100: 
> class fq_codel 100:3df parent 100: 
> 

[SM] They apparently have a rather loose translation from 80/10 GUI rates to 
shaper settings (derated by 5%), and my personal hobby horse seem to ignore the 
overhead question more or less entirely**... 
bad Ubiquity... if you want to be a "good boy" email me ;)


**) If that 5% derating is supposed to handle overhead, oh my... per packet 
overhead stays constant with packet number while payload size scales with 
packet size so overhead inversely scales with packet size and hence static 
rerating for per-packet overhead is approximate at best to stay polite.

Also what is up with the burst values? why do they differ by 32 octets?

Sorry for bothering you with this, these are mostly questions for Ubiquity and 
the are unlikely to monitor this list ;)

Regards
Sebastian



> On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller  wrote:
> Hi Dave.
> 
> just a few comments from the peanut gallery...
> 
> 
> > On Jan 2, 2024, at 19:59, dave seddon via Cake  
> > wrote:
> > 
> > G'day,
> > 
> > Happy new year y'all
> 
> +1
> 
> > 
> > I thought people might be interested to see what Ubiquity/Unifi is doing 
> > with "Smart Queues" on their devices.  The documentation on their website 
> > is not very informative.
> > 
> > Hopefully, this is vaguely interesting because Ubiquity is widely deployed 
> > and apparently they have a market cap of >$8 billion, so you would hope 
> > they do a "good job" (... Seems like they might be a target customer for 
> > libreqos )
> > 
> > 
> > https://finance.yahoo.com/quote/ui/
> > 
> > ( I use Unifi because their wifi stuff seems ok, and all the 
> > switching/routing/wifi is all integrated into the single gui control 
> > system.  Also honestly, I'm not sure I know how to do prefix delegation 
> > stuff on Linux by hand. )
> > 
> > Network diagram
> > 
> > Spectrum Cable Internets <--> Eth2 [ USG-Pro-4 ] Eth0 <---> 
> > [Switches] <> Access points
> > 
> > "Smart Queue" Configuration
> > Ubiquity doesn't have many knobs, you just enable "smart queues" and set 
> > the bandwidth.
> > 
> > 
> > 
> > 
> > "Smart Queue" Implementation
> > 
> > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT 
> > cake, but fq_codel.
> > 
> > And cake isn't available :(
> > 
> > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
> > Unknown qdisc "cake", hence option "bandwidth" is unparsable
> > 
> > Outbound eth2
> > 
> > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
> > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
> >  Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 
> > 0)  < OVERLIMITS?
> >  backlog 0b 0p requeues 0 
> > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 
> > 5.0ms interval 100.0ms ecn 
> >  Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0)  
> >  <- DROPS
> >  backlog 0b 0p requeues 0 
> >   maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
> >   new_flows_len 1 old_flows_len 1
> > qdisc ingress : parent :fff1  
> >  Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) 
> >  backlog 0b 0p requeues 0 
> >   • target 5.0ms is the default ( 
> > https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ).  I wonder 
> > if they did much testing on this hardware?
> 
> [SM] Not sure whether playing with target in isolation would be much use, in 
> codel theory target should be 5-10% of interval ans interval should be in the 
> order of magnitude of to be handled RTTs (the default is 100ms wich works 
> reasonably well even across the Atlantic, but you probably knew all that).
> 
> >   • ( I actually have a spare "wan" ethernet port, so I guess I 
> > could hook up a PC and perform a flent test. )
> >   • It's unclear to me what the "htb" is doing, because I would have 
> > expected the download/upload rates to be configured here, but they appear 
> > not to be
> 
> [SM] Likely because HTB does not reveal this when asked with the `-s` option, 
> try `-q` instead and not as qdisc but as class (so maybe `tc 

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[dave seddon via Cake]

Thanks Sebastian!

Now I see the rates!!

I actually reduced the rates to ensure this device is the bottleneck 80/10
Mb/s

[image: image.png]

root@USG-Pro-4:~# tc -d class show dev eth2
class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil
9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b
level 0
class fq_codel 100:12c parent 100:
class fq_codel 100:213 parent 100:
class fq_codel 100:22e parent 100:

root@USG-Pro-4:~# tc -d class show dev ifb_eth2
class htb 1:10 root leaf 100: prio 0 quantum 20 rate 76000Kbit ceil
76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead
0b level 0
class fq_codel 100:2c8 parent 100:
class fq_codel 100:3df parent 100:

On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller  wrote:

> Hi Dave.
>
> just a few comments from the peanut gallery...
>
>
> > On Jan 2, 2024, at 19:59, dave seddon via Cake <
> cake@lists.bufferbloat.net> wrote:
> >
> > G'day,
> >
> > Happy new year y'all
>
> +1
>
> >
> > I thought people might be interested to see what Ubiquity/Unifi is doing
> with "Smart Queues" on their devices.  The documentation on their website
> is not very informative.
> >
> > Hopefully, this is vaguely interesting because Ubiquity is widely
> deployed and apparently they have a market cap of >$8 billion, so you would
> hope they do a "good job" (... Seems like they might be a target customer
> for libreqos )
> >
> > 
> > https://finance.yahoo.com/quote/ui/
> >
> > ( I use Unifi because their wifi stuff seems ok, and all the
> switching/routing/wifi is all integrated into the single gui control
> system.  Also honestly, I'm not sure I know how to do prefix delegation
> stuff on Linux by hand. )
> >
> > Network diagram
> >
> > Spectrum Cable Internets <--> Eth2 [ USG-Pro-4 ] Eth0 <--->
> [Switches] <> Access points
> >
> > "Smart Queue" Configuration
> > Ubiquity doesn't have many knobs, you just enable "smart queues" and set
> the bandwidth.
> >
> >
> >
> >
> > "Smart Queue" Implementation
> >
> > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT
> cake, but fq_codel.
> >
> > And cake isn't available :(
> >
> > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
> > Unknown qdisc "cake", hence option "bandwidth" is unparsable
> >
> > Outbound eth2
> >
> > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
> > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver
> 3.17
> >  Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078
> requeues 0)  < OVERLIMITS?
> >  backlog 0b 0p requeues 0
> > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514
> target 5.0ms interval 100.0ms ecn
> >  Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues
> 0)   <- DROPS
> >  backlog 0b 0p requeues 0
> >   maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
> >   new_flows_len 1 old_flows_len 1
> > qdisc ingress : parent :fff1 
> >  Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues
> 0)
> >  backlog 0b 0p requeues 0
> >   • target 5.0ms is the default (
> https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ).  I wonder
> if they did much testing on this hardware?
>
> [SM] Not sure whether playing with target in isolation would be much use,
> in codel theory target should be 5-10% of interval ans interval should be
> in the order of magnitude of to be handled RTTs (the default is 100ms wich
> works reasonably well even across the Atlantic, but you probably knew all
> that).
>
> >   • ( I actually have a spare "wan" ethernet port, so I
> guess I could hook up a PC and perform a flent test. )
> >   • It's unclear to me what the "htb" is doing, because I would have
> expected the download/upload rates to be configured here, but they appear
> not to be
>
> [SM] Likely because HTB does not reveal this when asked with the `-s`
> option, try `-q` instead and not as qdisc but as class (so maybe `tc -d
> class show dev eth2`).
>
> >   • I'm not really sure what "overlimits" means or what that does,
> and tried looking this up, but I guess the kernel source is likely the
> "best" documentation for this.  Maybe this means it's dropping?  Or is it
> ECN?
>
> I think this text about TBF explains this reasonably well (HTB is
> essentially a hierarchical version of TBF):
>
> see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html
>
> 9.2.2. Token Bucket Filter
>
> The Token Bucket Filter (TBF) is a simple qdisc that only passes packets
> arriving at a rate which is not exceeding some administratively set rate,
> but with the possibility to allow short bursts in excess of this rate.
>
> TBF is very precise, network- and processor friendly. It should be your
> first choice if you simply want to slow an interface down!
>
> The TBF implementation consists of a buffer (bucket), constantly filled by
> some virtual pieces of 

Re: [Cake] Ubiquity (Unifi ) Smart Queues

[Sebastian Moeller via Cake]

Hi Dave.

just a few comments from the peanut gallery...


> On Jan 2, 2024, at 19:59, dave seddon via Cake  
> wrote:
> 
> G'day,
> 
> Happy new year y'all

+1

> 
> I thought people might be interested to see what Ubiquity/Unifi is doing with 
> "Smart Queues" on their devices.  The documentation on their website is not 
> very informative.
> 
> Hopefully, this is vaguely interesting because Ubiquity is widely deployed 
> and apparently they have a market cap of >$8 billion, so you would hope they 
> do a "good job" (... Seems like they might be a target customer for libreqos )
> 
> 
> https://finance.yahoo.com/quote/ui/
> 
> ( I use Unifi because their wifi stuff seems ok, and all the 
> switching/routing/wifi is all integrated into the single gui control system.  
> Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux 
> by hand. )
> 
> Network diagram
> 
> Spectrum Cable Internets <--> Eth2 [ USG-Pro-4 ] Eth0 <---> 
> [Switches] <> Access points
> 
> "Smart Queue" Configuration
> Ubiquity doesn't have many knobs, you just enable "smart queues" and set the 
> bandwidth.
> 
> 
> 
> 
> "Smart Queue" Implementation
> 
> Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, 
> but fq_codel.
> 
> And cake isn't available :(
> 
> root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
> Unknown qdisc "cake", hence option "bandwidth" is unparsable
> 
> Outbound eth2
> 
> root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
> qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
>  Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0)  
> < OVERLIMITS?
>  backlog 0b 0p requeues 0 
> qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 
> 5.0ms interval 100.0ms ecn 
>  Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0)
><- DROPS
>  backlog 0b 0p requeues 0 
>   maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
>   new_flows_len 1 old_flows_len 1
> qdisc ingress : parent :fff1  
>  Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0 
>   • target 5.0ms is the default ( 
> https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ).  I wonder if 
> they did much testing on this hardware?

[SM] Not sure whether playing with target in isolation would be much use, in 
codel theory target should be 5-10% of interval ans interval should be in the 
order of magnitude of to be handled RTTs (the default is 100ms wich works 
reasonably well even across the Atlantic, but you probably knew all that).

>   • ( I actually have a spare "wan" ethernet port, so I guess I 
> could hook up a PC and perform a flent test. )
>   • It's unclear to me what the "htb" is doing, because I would have 
> expected the download/upload rates to be configured here, but they appear not 
> to be

[SM] Likely because HTB does not reveal this when asked with the `-s` option, 
try `-q` instead and not as qdisc but as class (so maybe `tc -d class show dev 
eth2`).

>   • I'm not really sure what "overlimits" means or what that does, and 
> tried looking this up, but I guess the kernel source is likely the "best" 
> documentation for this.  Maybe this means it's dropping?  Or is it ECN?

I think this text about TBF explains this reasonably well (HTB is essentially a 
hierarchical version of TBF):

see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html

9.2.2. Token Bucket Filter

The Token Bucket Filter (TBF) is a simple qdisc that only passes packets 
arriving at a rate which is not exceeding some administratively set rate, but 
with the possibility to allow short bursts in excess of this rate.

TBF is very precise, network- and processor friendly. It should be your first 
choice if you simply want to slow an interface down!

The TBF implementation consists of a buffer (bucket), constantly filled by some 
virtual pieces of information called tokens, at a specific rate (token rate). 
The most important parameter of the bucket is its size, that is the number of 
tokens it can store.

Each arriving token collects one incoming data packet from the data queue and 
is then deleted from the bucket. Associating this algorithm with the two flows 
-- token and data, gives us three possible scenarios:

• The data arrives in TBF at a rate that's equal to the rate of 
incoming tokens. In this case each incoming packet has its matching token and 
passes the queue without delay.

• The data arrives in TBF at a rate that's smaller than the token rate. 
Only a part of the tokens are deleted at output of each data packet that's sent 
out the queue, so the tokens accumulate, up to the bucket size. The unused 
tokens can then be used to send data a a speed that's exceeding the standard 
token rate, in case short data bursts occur.

•