Hi Chris,
Thanks, OK taking your suggestion (I'm always interested in hearing
opinions and suggestions on how we can do things better) here is an
example of my queue logic (I've tried to keep it as short as possible
whilst providing full detail);
NB; We have a 100MBit leased line with 100 down and 100 up (in reality
when testing it really only achieves 95MBit up and 70Mbit down before
the Cisco CPE (or other upstream router) starts to mess with the packets.
The usable WAN upstream has been limited to 65MBit up, and the WAN
downstream is the sum of all the internal interfaces and is limited to
90Mbit (80+10). We have more interfaces than this (inc DMZ, WIFI etc and
the 90MBit downstream bandwidth is divided across all of them with more
granularity, but for this example I have included only 3 interfaces and
adjusted the figures to highlight the logic).
You'll notice that each interface has been split into two logical
bandwidth groups (e.g. lan_local and lan_wan), this is to ensure
inter-zone traffic (LAN->VoIP for example) does not use the WAN
bandwidth queues (uses available NIC bandwidth). And xxx_local_kernel is
used for guaranteeing carp packets no matter how busy the firewall is.
# EXAMPLE QUEUES
# EXT Local & WAN upstream queues
altq on $if_ext bandwidth 800Mb hfsc queue { ext_local, ext_wan }
queue ext_local bandwidth 700Mb priority 4 hfsc(upperlimit 700Mb) {
ext_local_kernel, ext_local_data }
queue ext_local_kernel bandwidth 1% qlimit 100 priority 4
hfsc(realtime 1%, linkshare 20%)
queue ext_local_data bandwidth 99% qlimit 100 priority 0
hfsc(linkshare 80%)
queue ext_wan bandwidth 65Mb priority 15 hfsc(linkshare 65Mb,
upperlimit 65Mb) { ext_wan_rt, ext_wan_int, ext_wan_pri, ext_wan_vpn,
ext_wan_web, ext_wan_dflt, ext_wan_bulk }
queue ext_wan_rt bandwidth 20% priority 15 qlimit 100
hfsc(realtime(35%, 5000, 20%), linkshare 20%)
queue ext_wan_int bandwidth 10% priority 14 qlimit 200
hfsc(realtime 5%, linkshare 10%)
queue ext_wan_pri bandwidth 20% priority 10 qlimit 200
hfsc(realtime(10%, 2000, 5%), linkshare 20%)
queue ext_wan_vpn bandwidth 10% priority 8 qlimit 300
hfsc(realtime 5%, linkshare 10%, ecn)
queue ext_wan_web bandwidth 10% priority 6 qlimit 500
hfsc(realtime(10%, 3000, 5%), linkshare 10%, ecn)
queue ext_wan_dflt bandwidth 20% priority 4 qlimit 100
hfsc(realtime(15%, 5000, 10%), linkshare 20%, ecn, default)
queue ext_wan_bulk bandwidth 5% priority 0 qlimit 100
hfsc(upperlimit 30%, linkshare 5%, ecn)
# LAN Local & WAN Downstream queues
altq on $if_lan bandwidth 800Mb hfsc queue { lan_local, lan_wan }
queue lan_local bandwidth 700Mb priority 4 hfsc(upperlimit 700Mb) {
lan_local_kernel, lan_local_data }
queue lan_local_kernel bandwidth 1% qlimit 100 priority 4
hfsc(realtime 1%, linkshare 20%)
queue lan_local_data bandwidth 99% qlimit 100 priority 0
hfsc(linkshare 80%)
queue lan_wan bandwidth 80Mb priority 15 hfsc(linkshare 80Mb,
upperlimit 80Mb) { lan_wan_rt, lan_wan_int, lan_wan_pri, lan_wan_vpn,
lan_wan_web, lan_wan_dflt, lan_wan_bulk }
queue lan_wan_rt bandwidth 20% priority 15 qlimit 100
hfsc(realtime(30%, 5000, 15%), linkshare 20%)
queue lan_wan_int bandwidth 10% priority 14 qlimit 200
hfsc(realtime 5%, linkshare 10%)
queue lan_wan_pri bandwidth 10% priority 10 qlimit 300
hfsc(realtime(10%, 2000, 5%), linkshare 10%)
queue lan_wan_vpn bandwidth 10% priority 8 qlimit 300
hfsc(realtime 5%, linkshare 10%, ecn)
queue lan_wan_web bandwidth 20% priority 6 qlimit 500
hfsc(realtime(15%, 3000, 5%), linkshare 20%, ecn)
queue lan_wan_dflt bandwidth 20% priority 4 qlimit 100
hfsc(realtime(15%, 5000, 10%), linkshare 20%, ecn, default)
queue lan_wan_bulk bandwidth 5% priority 0 qlimit 100
hfsc(upperlimit 30%, linkshare 5%, ecn)
# VoIP Local & WAN Downstream queues
altq on $if_voip bandwidth 800Mb hfsc queue { voip_local, voip_wan }
queue voip_local bandwidth 700Mb priority 4 hfsc(upperlimit 700Mb)
{ voip_local_kernel, voip_local_data }
queue voip_local_kernel bandwidth 1% qlimit 100 priority 4
hfsc(realtime 1%, linkshare 20%)
queue voip_local_data bandwidth 99% qlimit 100 priority 0
hfsc(linkshare 80%)
queue voip_wan bandwidth 10Mb priority 15 hfsc(linkshare 10Mb,
upperlimit 10Mb) { voip_wan_rt, voip_wan_pri, voip_wan_dflt }
queue voip_wan_rt bandwidth 50% priority 15 qlimit 100
hfsc(realtime(60%, 5000, 40%), linkshare 50%)
queue voip_wan_pri bandwidth 20% priority 10 qlimit 300
hfsc(realtime(20%, 3000, 10%), linkshare 20%)
queue voip_wan_dflt bandwidth 10% priority 2 qlimit 100
hfsc(upperlimit 50%, linkshare 10%, ecn, default)
# EXAMPLE RULES
# EXT Interface
pass out on $if_ext all modulate state (pflow) queue
(ext_wan_dflt,ext_wan_pri) # Default out (all) - queues for WAN upstream
# No user traffic rule for inter-zone-traffic on EXT WAN interface
pass quick on $if_ext proto tcp from $ext_net_wan to any port {
$interactive_tcpports } queue (ext_wan_bulk,ext_wan_int) # Outbound
SSH low latency for key strokes and low priority for file transfers -
queues for WAN upstream
pass quick on $if_ext proto { tcp, udp } from { <ext_voipgateway_nets> }
to { $int_ip_noise1 } port { 4569, 5060, 10000:20000 } queue
ext_wan_rt # Inbound VoIP - queues for WAN upstream
# LAN Interface
pass in on $if_lan from { $int_net_lan } to any queue
(lan_wan_dflt,lan_wan_pri) # Default out (lan) - queues for
WAN downstream
pass quick on $if_lan from { $int_net_lan } to { $int_net_voip } queue
lan_local_data # Ensure inter-zone-traffic uses lan_local queues
pass quick on $if_lan proto tcp from { $int_net_lan } to any port {
$interactive_tcpports } queue (lan_wan_bulk,lan_wan_int) # Outbound
SSH low latency for key strokes and low priority for file transfers -
queues for WAN downstream
# VoIP Interface
pass in on $if_voip from { $int_net_voip } to any queue
(voip_wan_dflt,voip_wan_pri) # Default out (voip) - queues for
WAN downstream
pass quick on $if_voip from { $int_net_lan } to { $int_net_voip } queue
voip_local_data # Ensure inter-zone-traffic uses voip_local queues
pass quick on $if_voip proto { tcp, udp } from { <ext_voipgateway_nets>
} to { $int_ip_noise1 } port { 4569, 5060, 10000:20000 } queue
voip_wan_rt # Inbound VoIP - queues for WAN downstream
In these example rules (always in pairs)
1- First you can see that I have a default traffic out (upstream) on the
EXT interface (with queues for WAN egress based on WAN upstream
bandwidth), a default traffic out (upstream) on the LAN interface (going
'into' the LAN interface, but with queues for LAN egress (part of WAN
downstream bandwidth)), and another default traffic out (upstream) on
the VoIP interface (going 'into' the VoIP interface, but with queues for
VoIP egress (other part of WAN downstream bandwidth)).
2- Next you can see an example rule for allowing traffic going from the
LAN to the VoIP network ingressing the LAN interface (but with queues
for LAN egress based on NIC bandwidth), and a rule allowing traffic
going from the LAN to the VoIP network egressing the VoIP interface
(with queues for VoIP egress).
3- Next you can see an example rule allowing outbound initiated SSH
traffic from the LAN to the WAN ingressing the LAN interface (with
queues for LAN egress based on WAN bandwidth), and a rule allowing
outbound initiated SSH traffic from the LAN to the WAN egressing the EXT
interface (with queues for EXT egress). NB; Notice that the queues used
here have lower priority for SSH bulk (file transfers), but higher
priority for key strokes compared to the default out rules.
4- And finally you can also see an example rule pair for inbound
initiated VoIP connections from our VoIP providers, with a rule allowing
WAN to VoIP ingressing EXT (with queues for EXT egress (based on WAN
upstream)), and a rule allowing WAN to VoIP egressing the VoIP interface
(with queues for VoIP egress (based on WAN downstream)).
* Whilst it takes a little getting used too, you just need to remember
some key rules; when a SYN packet matches a rule, 'state' is created and
if a queue statement exists on the rule, the queue name is added to the
state.
When a subsequent packet which is part of the connection matches a state
(entering or leaving the firewall), pf will read the queue name stored
with that state, and IF a queue of the same name exists as the packet
'egress' the firewall, it will be queued in that queue.
So whilst it may seem strange to create state as the packet enters an
interface and record a queue name in the state which only works in the
opposite (egress) direction (ALTQ can only queue on egress), all 'reply'
packets which are part of the connection will get queued as they egress
the internal interface.
These rules allow the phones to work perfectly, and SSH key strokes to
be fast (even when running SCP), even when every single person in the
company is downloading CD's & DVD's etc.
The only downside is that it is not possible to have VoIP access to the
full 90MBit whilst the LAN is quiet, or for the LAN to have the full
90MBit whilst the VoIP is quiet etc. Both LAN and VoIP can dynamically
share the 60Mbit upload bandwidth.
Their is so much more to say here but in the interests of keeping this
short (it is already very long) I will leave it here. I'm happy to say
more about HFSC best practices or anything else if anyone is interested.
Thanks for your time and reading this far, Andrew Lemin
On 03/06/13 17:43, Chris Cappuccio wrote:
Andy [a...@brandwatch.com] wrote:
Hi,
We're really looking forward to improvements in ALTQ too.
And we are /really/ hoping that the queues can either be shared across
interfaces (so your WAN downstream bandwidth doesn't have to be sliced
up and divided up across all the internal interfaces), or that you can
create queues on the external interface's 'ingress' flow.
I know this opens a can of worms as many say you can't theoretically
shape inbound bandwidth as you've already received the packets, however
we do shape inbound bandwidth and it works brilliantly! But you have to
do it on each of the internal interfaces egress (hence having to slice
up the total downstream), so connections receiving too many downstream
packets are slowed by dropping some of the already received TCP packets
(not perfect but it works).
You should post your ruleset. It sounds like you may be able to get some
better performance without new functionality.
Also whilst I'm wishing, also looking forward to the day that the
FQ_Codel algorithms etc which significantly improve buffer-bloat are
soon in OpenBSD (now in Linux 3.7 :)
Honestly, who cares about buffer bloat? Just because it's a
popular issue in some circles does not mean that anything you do
on your openbsd firewall is going to affect the problem one way or
another.
