Hello list members. Long time no see. Law school hasn't been kind to my free
time.
I have noticed this unexpected behavior in pf since OpenBSD 5.0-release on
various amd64 boxes. I am using cbq in pf to throttle outbound traffic on both
interfaces of my home router. Full pf rule set and dmesg appear at the end of
this email.
The external interface on the router is connected to ADSL with total provided
bandwidth of approximately 5500Kb. The internal interface is an em on a 1Gb
network.
Compare these two blocks of pf queue configuration:
# Block #1
altq on $IntIf cbq bandwidth 5250Kb queue \
{ LGO2X_DL, PC_DL, Skype_DL, TV_DL, WiFi_DL }
queue LGO2X_DL bandwidth 650Kb priority 2 cbq(borrow ecn)
queue PC_DL bandwidth 1500Kb priority 1 cbq(borrow ecn default)
queue Skype_DL bandwidth 100Kb priority 2 cbq(ecn)
queue TV_DL bandwidth 2000Kb priority 1 cbq(borrow ecn)
queue WiFi_DL bandwidth 1000Kb priority 1 cbq(borrow ecn)
# Block #2
altq on $IntIf cbq bandwidth 5253Kb queue \
{ PARENT }
queue PARENT bandwidth 5250Kb \
{ LGO2X_DL, PC_DL, Skype_DL, TV_DL, WiFi_DL }
queue LGO2X_DL bandwidth 650Kb priority 2 cbq(borrow ecn)
queue PC_DL bandwidth 1500Kb priority 1 cbq(borrow ecn default)
queue Skype_DL bandwidth 100Kb priority 2 cbq(ecn)
queue TV_DL bandwidth 2000Kb priority 1 cbq(borrow ecn)
queue WiFi_DL bandwidth 1000Kb priority 1 cbq(borrow ecn)
Block #1 performs as expected. The queues receive their guaranteed minimum
allotment, and can borrow bandwidth up to the maximum allotted to the
interface.
One might expect the queues under PARENT in block #2 to perform in the same
fashion. They do not.
- When the bandwidth allocation for the interface is more than 3Kb
greater than the bandwidth allocation for the PARENT queue, borrowing by the
child queues is degraded significantly. Given block #2 above, the following
performance was noted:
o With interface bandwidth allocated at 5253Kb the child queues only borrow
to approximately 3200Kbps;
o With interface bandwidth allocated at 1Gb the child queues only borrow to
approximately 2600Kbps.
- When the bandwidth allocation for the interface is between 5250Kb
and 5252Kb the child queues perform as they do using the configuration in
block #1.
- When cbq(borrow) is configured on the parent queue the child queues
can achieve the maximum bandwidth of the external interface.
This configuration was also tested:
#Block 3
altq on $IntIf cbq bandwidth 5250Kb queue \
{ PARENT }
queue PARENT bandwidth 2500Kb \
{ LGO2X_DL, PC_DL, Skype_DL, TV_DL, WiFi_DL }
queue LGO2X_DL bandwidth 500Kb priority 2 cbq(borrow ecn)
queue PC_DL bandwidth 500Kb priority 1 cbq(borrow ecn default)
queue Skype_DL bandwidth 500Kb priority 2 cbq(ecn)
queue TV_DL bandwidth 500Kb priority 1 cbq(borrow ecn)
queue WiFi_DL bandwidth 500Kb priority 1 cbq(borrow ecn)
In this case, the child queues would not borrow past 650Kb total bandwidth.
The anticipated behavior was a borrow to 2500Kb.
Is there an explanation for this behavior, other than an error in pf itself?
This outcome does not seem to follow from the documentation, neither with the
man pages nor the pf FAQ.
Thanks.
Breen Ouellette
Full pf rule set:
# PF optimized for home router.
# UPDATED: 2012-Nov-17
###########
# Preamble:
####
# REMEMBER: Enable the following line in /etc/sysctl.conf:
# net.inet.ip.forwarding=1
####
# <Blocked1Hr> table requires matching crontab entry to expire blocked IPs:
# * * * * * pfctl -t Blocked1Hr -T expire 3600 >
/dev/null 2>&1
####
# Filtering of ssh abusers also requires that sshd_config is updated to listen
on ports 10 and 16.
ExtIf = "em1"
ExtIP = "(em1)"
IntIf = "em0"
VLAN1If = "vlan1"
VLAN1Net = "vlan1:network"
VLAN2If = "vlan2"
VLAN2Net = "vlan2:network"
AthenaIP = "172.16.0.1"
ScreenerIP = "172.16.0.2"
LGO2XIP = "192.168.0.100"
SkypeIP = "192.168.0.50"
table <authpf_users> persist
table <Blocked1Hr> persist
table <SSHBlockedOnce> persist
table <SSHBlockedTwice> persist
table <SSHBlockedPerm> persist
table <CdnNets> const persist file "/etc/cdn_nets.pftable"
table <MartianNets> const persist file "/etc/martian_nets.pftable"
table <SSHAllowedIPs> const persist file "/etc/ssh_ips.pftable"
# Internal network queuing.
altq on $IntIf cbq bandwidth 5250Kb queue \
{ LGO2X_DL, PC_DL, Skype_DL, TV_DL, WiFi_DL }
queue LGO2X_DL bandwidth 650Kb priority 2 cbq(borrow ecn)
queue PC_DL bandwidth 1500Kb priority 1 cbq(borrow ecn default)
queue Skype_DL bandwidth 100Kb priority 2 cbq(ecn)
queue TV_DL bandwidth 2000Kb priority 1 cbq(borrow ecn)
queue WiFi_DL bandwidth 1000Kb priority 1 cbq(borrow ecn)
# ISP network queuing.
altq on $ExtIf cbq bandwidth 550Kb queue \
{ ACK_UL, LGO2X_UL, PC_UL, Skype_UL, TV_UL, WiFi_UL }
# 27400bps ACK_UL is required for each 1Mbps of total download bandwitdh,
# assuming a 40bit ACK packet size. Real world results may vary and
values
# used here were derived through experimentation.
# Too much ACK bandwidth is better than not enough, given borrow
# relationships.
queue ACK_UL bandwidth 176Kb priority 7 cbq(ecn)
queue LGO2X_UL bandwidth 64Kb priority 1 cbq(borrow ecn)
queue PC_UL bandwidth 82Kb priority 0 cbq(borrow ecn default)
queue Skype_UL bandwidth 100Kb priority 1 cbq(ecn)
queue TV_UL bandwidth 64Kb priority 0 cbq(borrow ecn)
queue WiFi_UL bandwidth 64Kb priority 0 cbq(borrow ecn)
set skip on lo
set state-policy if-bound
block
block quick inet6
block in quick from <SSHBlockedPerm>
block out quick to <SSHBlockedPerm>
block in quick on $ExtIf from <MartianNets>
block out quick on $ExtIf to <MartianNets>
block in quick on $ExtIf from <Blocked1Hr>
block out quick on $ExtIf to <Blocked1Hr>
block in quick proto tcp from <SSHBlockedOnce> to any port ssh
block in quick proto tcp from <SSHBlockedTwice> to any port 16
match out on $ExtIf inet to ! <MartianNets> \
nat-to $ExtIP port 1024:65535 received-on $VLAN1If
match out on $ExtIf inet to ! <MartianNets> \
nat-to $ExtIP port 1024:65535 received-on $VLAN2If
pass in quick on $VLAN1If inet proto udp \
from any port bootpc to { 255.255.255.255/32, $VLAN1If } port bootps
pass in quick on $VLAN1If inet proto udp \
from $AthenaIP to $VLAN1If port ntp
pass in quick on $VLAN1If inet proto tcp \
from $VLAN1Net to $VLAN1If port ssh keep state \
(max-src-conn 16, max-src-conn-rate 8/300, overload <SSHBlockedPerm>) \
queue(PC_DL) \
tag SSH_INTERNAL
pass in quick on $VLAN1If inet \
from $ScreenerIP to ! <MartianNets> \
queue(TV_DL) \
tag TV_PACKET
pass in quick on $VLAN1If inet \
from $VLAN1Net to ! <MartianNets> \
queue(PC_DL) \
tag PC_PACKET
pass out quick on $VLAN1If inet \
queue(TV_DL) \
tagged TV_PACKET
pass out quick on $VLAN2If inet \
queue(PC_DL) \
tagged SSH_INTERNAL
pass out quick on $VLAN1If inet \
from $VLAN1If \
queue(PC_DL)
pass in quick on $VLAN2If inet proto udp \
from any port bootpc to { 255.255.255.255/32, $IntIf } port bootps
pass in quick on $VLAN2If inet proto tcp \
from $VLAN2Net to $VLAN2If port ssh keep state \
(max-src-conn 16, max-src-conn-rate 8/300, overload <SSHBlockedPerm>) \
queue(WiFi_DL) \
tag SSH_INTERNAL
pass in quick on $VLAN2If inet \
from $LGO2XIP to ! <MartianNets> \
queue(LGO2X_DL) \
tag LGO2X_PACKET
pass in quick on $VLAN2If inet \
from $SkypeIP to ! <MartianNets> \
queue(Skype_DL) \
tag SKYPE_PACKET
pass in quick on $VLAN2If inet \
from <authpf_users> to ! <MartianNets> \
queue(WiFi_DL) \
tag WIFI_PACKET
pass out quick on $VLAN2If inet \
queue(WiFi_DL) \
tagged SSH_INTERNAL
pass out quick on $VLAN2If inet \
from $VLAN2If \
queue(WiFi_DL)
pass in quick on $ExtIf inet proto tcp \
from ! <MartianNets> to $ExtIP port 65152 rdr-to $ScreenerIP keep state \
(max-src-conn-rate 4/60, tcp.established 60, overload <Blocked1Hr>) \
queue(TV_UL, ACK_UL) \
tag TV_PACKET
pass in quick on $ExtIf inet proto tcp \
from <CdnNets> to $ExtIP port ssh keep state \
(max-src-conn 16, max-src-conn-rate 8/300, overload <SSHBlockedOnce>) \
queue(PC_UL, ACK_UL)
pass in quick on $ExtIf inet proto tcp \
from <SSHIPs> to $ExtIP port ssh keep state \
(max-src-conn 16, max-src-conn-rate 8/300, overload <SSHBlockedOnce>) \
queue(PC_UL, ACK_UL)
pass in quick on $ExtIf inet proto tcp \
from <SSHBlockedOnce> to $ExtIP port 16 keep state \
(max-src-conn 12, max-src-conn-rate 6/900, overload <SSHBlockedTwice>) \
queue(PC_UL, ACK_UL)
pass in quick on $ExtIf inet proto tcp \
from <SSHBlockedTwice> to $ExtIP port 10 keep state \
(max-src-conn 8, max-src-conn-rate 4/3600, overload <SSHBlockedPerm>) \
queue(PC_UL, ACK_UL)
pass out quick on $ExtIf inet \
queue(TV_UL, ACK_UL) \
tagged TV_PACKET
pass out quick on $ExtIf inet \
queue(PC_UL, ACK_UL) \
tagged PC_PACKET
pass out quick on $ExtIf inet \
queue(LGO2X_UL, ACK_UL) \
tagged LGO2X_PACKET
pass out quick on $ExtIf inet \
queue(Skype_UL, ACK_UL) \
tagged SKYPE_PACKET
pass out quick on $ExtIf inet \
queue(WiFi_UL, ACK_UL) \
tagged WIFI_PACKET
pass out quick on $ExtIf inet \
from $ExtIP \
queue(PC_UL, ACK_UL)
# EOF: /etc/pf.conf
dmesg:
OpenBSD 5.2 (GENERIC.MP) #368: Wed Aug 1 10:04:49 MDT 2012
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP<mail
to:dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP>
real mem = 2145910784 (2046MB)
avail mem = 2066477056 (1970MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x9f000 (19 entries)
bios0: vendor American Megatrends Inc. version "1.1a" date 12/17/10
bios0: Supermicro X7SPA-HF
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB HPET EINJ BERT ERST HEST
acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4)
USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) USBE(S4) P0P4(S4) P0P5(S4)
P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) GBE_(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.27 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xT
PR,PDCM,MOVBE,NXE,LONG,LAHF
cpu0: 512KB 64b/line 8-way L2 cache
cpu0: apic clock running at 201MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1818.00 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xT
PR,PDCM,MOVBE,NXE,LONG,LAHF
cpu1: 512KB 64b/line 8-way L2 cache
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1818.00 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xT
PR,PDCM,MOVBE,NXE,LONG,LAHF
cpu2: 512KB 64b/line 8-way L2 cache
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1818.00 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xT
PR,PDCM,MOVBE,NXE,LONG,LAHF
cpu3: 512KB 64b/line 8-way L2 cache
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 1, remapped to apid 4
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P0P1)
acpiprt2 at acpi0: bus 1 (P0P4)
acpiprt3 at acpi0: bus -1 (P0P5)
acpiprt4 at acpi0: bus -1 (P0P6)
acpiprt5 at acpi0: bus -1 (P0P7)
acpiprt6 at acpi0: bus 2 (P0P8)
acpiprt7 at acpi0: bus 3 (P0P9)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 4 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 4 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 4 int 19
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 4 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: msi
pci2 at ppb1 bus 2
em0 at pci2 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:37:34:3e
ppb2 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: msi
pci3 at ppb2 bus 3
em1 at pci3 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:37:34:3f
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 4 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 4 int 19
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 4 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 4 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci4 at ppb3 bus 4
vga1 at pci4 dev 4 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, INTEL SSDSA2CT04, 4PC1> SCSI3 0/direct
fixed naa.5001517959516592
sd0: 38166MB, 512 bytes/sector, 78165360 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 4 int
18
iic0 at ichiic0
lm1 at iic0 addr 0x2d: W83627DHG
spdmem0 at iic0 addr 0x51: 2GB DDR3 SDRAM PC3-10600 SO-DIMM
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627DHG rev 0x25
lm2 at wbsio0 port 0xca0/8: W83627DHG
mtrr: Pentium Pro MTRR support
lm1: disabling sensors due to alias with lm2
uhidev0 at uhub4 port 2 configuration 1 interface 0 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev1 at uhub4 port 2 configuration 1 interface 1 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev2 at uhub5 port 1 configuration 1 interface 0 "Lite-On Technology Corp.
HP Basic USB Keyboard" rev 1.10/1.03 addr 2
uhidev2: iclass 3/1
ukbd1 at uhidev2: 8 variable keys, 6 key codes
wskbd2 at ukbd1 mux 1
wskbd2: connecting to wsdisplay0
uhidev3 at uhub5 port 2 configuration 1 interface 0 "Logitech Logitech USB
Optical Mouse" rev 2.00/43.00 addr 3
uhidev3: iclass 3/1
ums1 at uhidev3: 3 buttons, Z dir
wsmouse1 at ums1 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (8ea4ab417a9972a0.a) swap on sd0b dump on sd0b
