Re: [Cake] Ubiquity (Unifi ) Smart Queues
> On 2 Jan, 2024, at 8:59 pm, dave seddon via Cake > wrote: > > • I'm not really sure what "overlimits" means or what that does, and > tried looking this up, but I guess the kernel source is likely the "best" > documentation for this. Maybe this means it's dropping? Or is it ECN? Overlimit just means the shaper in HTB is restricting the flow, thus doing something useful. Drop means the AQM is working to provide congestion information to a Not-ECT flow. Those numbers look normal (about 0.2% drop rate, and most packets hitting "overlimit" when the link is saturated). Using fq_codel's default parameters is not a bad thing at all. I'd much rather they did that, thereby using numbers that are tuned for general Internet conditions, than try to change that tuning in ignorance of the end-user's actual network environment. Most end-users have their WAN port facing "general Internet conditions" anyway. > Apparently rather than applying the tc qdsic on the outbound path on the LAN > side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2. That's the correct place to do it, so that the qdisc applies specifically to traffic coming from the WAN. If you apply it to the LAN egress, it tends to affect traffic coming through the router from the WiFi or its internal servers, which is not desirable. These are all questions we've considered at length in the process of developing and deploying Cake and other SQM solutions. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Ubiquity (Unifi ) Smart Queues
Hi Dave, > On Jan 2, 2024, at 22:15, dave seddon wrote: > > Thanks Sebastian! > > Now I see the rates!! > > I actually reduced the rates to ensure this device is the bottleneck 80/10 > Mb/s > > > > root@USG-Pro-4:~# tc -d class show dev eth2 > class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil > 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b > level 0 > class fq_codel 100:12c parent 100: > class fq_codel 100:213 parent 100: > class fq_codel 100:22e parent 100: > > root@USG-Pro-4:~# tc -d class show dev ifb_eth2 > class htb 1:10 root leaf 100: prio 0 quantum 20 rate 76000Kbit ceil > 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b > level 0 > class fq_codel 100:2c8 parent 100: > class fq_codel 100:3df parent 100: > [SM] They apparently have a rather loose translation from 80/10 GUI rates to shaper settings (derated by 5%), and my personal hobby horse seem to ignore the overhead question more or less entirely**... bad Ubiquity... if you want to be a "good boy" email me ;) **) If that 5% derating is supposed to handle overhead, oh my... per packet overhead stays constant with packet number while payload size scales with packet size so overhead inversely scales with packet size and hence static rerating for per-packet overhead is approximate at best to stay polite. Also what is up with the burst values? why do they differ by 32 octets? Sorry for bothering you with this, these are mostly questions for Ubiquity and the are unlikely to monitor this list ;) Regards Sebastian > On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller wrote: > Hi Dave. > > just a few comments from the peanut gallery... > > > > On Jan 2, 2024, at 19:59, dave seddon via Cake > > wrote: > > > > G'day, > > > > Happy new year y'all > > +1 > > > > > I thought people might be interested to see what Ubiquity/Unifi is doing > > with "Smart Queues" on their devices. The documentation on their website > > is not very informative. > > > > Hopefully, this is vaguely interesting because Ubiquity is widely deployed > > and apparently they have a market cap of >$8 billion, so you would hope > > they do a "good job" (... Seems like they might be a target customer for > > libreqos ) > > > > > > https://finance.yahoo.com/quote/ui/ > > > > ( I use Unifi because their wifi stuff seems ok, and all the > > switching/routing/wifi is all integrated into the single gui control > > system. Also honestly, I'm not sure I know how to do prefix delegation > > stuff on Linux by hand. ) > > > > Network diagram > > > > Spectrum Cable Internets <--> Eth2 [ USG-Pro-4 ] Eth0 <---> > > [Switches] <> Access points > > > > "Smart Queue" Configuration > > Ubiquity doesn't have many knobs, you just enable "smart queues" and set > > the bandwidth. > > > > > > > > > > "Smart Queue" Implementation > > > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT > > cake, but fq_codel. > > > > And cake isn't available :( > > > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms > > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > > Outbound eth2 > > > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 > > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 > > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues > > 0) < OVERLIMITS? > > backlog 0b 0p requeues 0 > > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target > > 5.0ms interval 100.0ms ecn > > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) > > <- DROPS > > backlog 0b 0p requeues 0 > > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 > > new_flows_len 1 old_flows_len 1 > > qdisc ingress : parent :fff1 > > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) > > backlog 0b 0p requeues 0 > > • target 5.0ms is the default ( > > https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder > > if they did much testing on this hardware? > > [SM] Not sure whether playing with target in isolation would be much use, in > codel theory target should be 5-10% of interval ans interval should be in the > order of magnitude of to be handled RTTs (the default is 100ms wich works > reasonably well even across the Atlantic, but you probably knew all that). > > > • ( I actually have a spare "wan" ethernet port, so I guess I > > could hook up a PC and perform a flent test. ) > > • It's unclear to me what the "htb" is doing, because I would have > > expected the download/upload rates to be configured here, but they appear > > not to be > > [SM] Likely because HTB does not reveal this when asked with the `-s` option, > try `-q` instead and not as qdisc but as class (so maybe `tc
Re: [Cake] Ubiquity (Unifi ) Smart Queues
Thanks Sebastian! Now I see the rates!! I actually reduced the rates to ensure this device is the bottleneck 80/10 Mb/s [image: image.png] root@USG-Pro-4:~# tc -d class show dev eth2 class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b level 0 class fq_codel 100:12c parent 100: class fq_codel 100:213 parent 100: class fq_codel 100:22e parent 100: root@USG-Pro-4:~# tc -d class show dev ifb_eth2 class htb 1:10 root leaf 100: prio 0 quantum 20 rate 76000Kbit ceil 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b level 0 class fq_codel 100:2c8 parent 100: class fq_codel 100:3df parent 100: On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller wrote: > Hi Dave. > > just a few comments from the peanut gallery... > > > > On Jan 2, 2024, at 19:59, dave seddon via Cake < > cake@lists.bufferbloat.net> wrote: > > > > G'day, > > > > Happy new year y'all > > +1 > > > > > I thought people might be interested to see what Ubiquity/Unifi is doing > with "Smart Queues" on their devices. The documentation on their website > is not very informative. > > > > Hopefully, this is vaguely interesting because Ubiquity is widely > deployed and apparently they have a market cap of >$8 billion, so you would > hope they do a "good job" (... Seems like they might be a target customer > for libreqos ) > > > > > > https://finance.yahoo.com/quote/ui/ > > > > ( I use Unifi because their wifi stuff seems ok, and all the > switching/routing/wifi is all integrated into the single gui control > system. Also honestly, I'm not sure I know how to do prefix delegation > stuff on Linux by hand. ) > > > > Network diagram > > > > Spectrum Cable Internets <--> Eth2 [ USG-Pro-4 ] Eth0 <---> > [Switches] <> Access points > > > > "Smart Queue" Configuration > > Ubiquity doesn't have many knobs, you just enable "smart queues" and set > the bandwidth. > > > > > > > > > > "Smart Queue" Implementation > > > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT > cake, but fq_codel. > > > > And cake isn't available :( > > > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms > > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > > Outbound eth2 > > > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 > > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver > 3.17 > > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 > requeues 0) < OVERLIMITS? > > backlog 0b 0p requeues 0 > > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 > target 5.0ms interval 100.0ms ecn > > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues > 0) <- DROPS > > backlog 0b 0p requeues 0 > > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 > > new_flows_len 1 old_flows_len 1 > > qdisc ingress : parent :fff1 > > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues > 0) > > backlog 0b 0p requeues 0 > > • target 5.0ms is the default ( > https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder > if they did much testing on this hardware? > > [SM] Not sure whether playing with target in isolation would be much use, > in codel theory target should be 5-10% of interval ans interval should be > in the order of magnitude of to be handled RTTs (the default is 100ms wich > works reasonably well even across the Atlantic, but you probably knew all > that). > > > • ( I actually have a spare "wan" ethernet port, so I > guess I could hook up a PC and perform a flent test. ) > > • It's unclear to me what the "htb" is doing, because I would have > expected the download/upload rates to be configured here, but they appear > not to be > > [SM] Likely because HTB does not reveal this when asked with the `-s` > option, try `-q` instead and not as qdisc but as class (so maybe `tc -d > class show dev eth2`). > > > • I'm not really sure what "overlimits" means or what that does, > and tried looking this up, but I guess the kernel source is likely the > "best" documentation for this. Maybe this means it's dropping? Or is it > ECN? > > I think this text about TBF explains this reasonably well (HTB is > essentially a hierarchical version of TBF): > > see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html > > 9.2.2. Token Bucket Filter > > The Token Bucket Filter (TBF) is a simple qdisc that only passes packets > arriving at a rate which is not exceeding some administratively set rate, > but with the possibility to allow short bursts in excess of this rate. > > TBF is very precise, network- and processor friendly. It should be your > first choice if you simply want to slow an interface down! > > The TBF implementation consists of a buffer (bucket), constantly filled by > some virtual pieces of
Re: [Cake] Ubiquity (Unifi ) Smart Queues
Hi Dave. just a few comments from the peanut gallery... > On Jan 2, 2024, at 19:59, dave seddon via Cake > wrote: > > G'day, > > Happy new year y'all +1 > > I thought people might be interested to see what Ubiquity/Unifi is doing with > "Smart Queues" on their devices. The documentation on their website is not > very informative. > > Hopefully, this is vaguely interesting because Ubiquity is widely deployed > and apparently they have a market cap of >$8 billion, so you would hope they > do a "good job" (... Seems like they might be a target customer for libreqos ) > > > https://finance.yahoo.com/quote/ui/ > > ( I use Unifi because their wifi stuff seems ok, and all the > switching/routing/wifi is all integrated into the single gui control system. > Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux > by hand. ) > > Network diagram > > Spectrum Cable Internets <--> Eth2 [ USG-Pro-4 ] Eth0 <---> > [Switches] <> Access points > > "Smart Queue" Configuration > Ubiquity doesn't have many knobs, you just enable "smart queues" and set the > bandwidth. > > > > > "Smart Queue" Implementation > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, > but fq_codel. > > And cake isn't available :( > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > Outbound eth2 > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) > < OVERLIMITS? > backlog 0b 0p requeues 0 > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target > 5.0ms interval 100.0ms ecn > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) ><- DROPS > backlog 0b 0p requeues 0 > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 > new_flows_len 1 old_flows_len 1 > qdisc ingress : parent :fff1 > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > • target 5.0ms is the default ( > https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder if > they did much testing on this hardware? [SM] Not sure whether playing with target in isolation would be much use, in codel theory target should be 5-10% of interval ans interval should be in the order of magnitude of to be handled RTTs (the default is 100ms wich works reasonably well even across the Atlantic, but you probably knew all that). > • ( I actually have a spare "wan" ethernet port, so I guess I > could hook up a PC and perform a flent test. ) > • It's unclear to me what the "htb" is doing, because I would have > expected the download/upload rates to be configured here, but they appear not > to be [SM] Likely because HTB does not reveal this when asked with the `-s` option, try `-q` instead and not as qdisc but as class (so maybe `tc -d class show dev eth2`). > • I'm not really sure what "overlimits" means or what that does, and > tried looking this up, but I guess the kernel source is likely the "best" > documentation for this. Maybe this means it's dropping? Or is it ECN? I think this text about TBF explains this reasonably well (HTB is essentially a hierarchical version of TBF): see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html 9.2.2. Token Bucket Filter The Token Bucket Filter (TBF) is a simple qdisc that only passes packets arriving at a rate which is not exceeding some administratively set rate, but with the possibility to allow short bursts in excess of this rate. TBF is very precise, network- and processor friendly. It should be your first choice if you simply want to slow an interface down! The TBF implementation consists of a buffer (bucket), constantly filled by some virtual pieces of information called tokens, at a specific rate (token rate). The most important parameter of the bucket is its size, that is the number of tokens it can store. Each arriving token collects one incoming data packet from the data queue and is then deleted from the bucket. Associating this algorithm with the two flows -- token and data, gives us three possible scenarios: • The data arrives in TBF at a rate that's equal to the rate of incoming tokens. In this case each incoming packet has its matching token and passes the queue without delay. • The data arrives in TBF at a rate that's smaller than the token rate. Only a part of the tokens are deleted at output of each data packet that's sent out the queue, so the tokens accumulate, up to the bucket size. The unused tokens can then be used to send data a a speed that's exceeding the standard token rate, in case short data bursts occur. •
