Forgive me for the long mail. I went out of my way to be thorough because I see mails like this on misc@ fairly often. I had the same kinds of questions when I set up my OpenBSD router years ago so I can empathize.
> My question are: > > 1. For better utilization of TCP traffic I have two priorities assigned to > the queue. Do I require more than one sub queue for this to work ? I > don't intend to subdivide my traffic up (i.e. a SSH queue, and HTTP/S > queue, etc.), I just want all my TCP traffic to benefit from better > utilization with the two priorities. No, and actually I'm pretty sure you can go without the sub queues entirely. Just one queue is enough and will work the way you want it to. > 2. If this configuration is currently correct, are they any other changes > I should make for better queuing (ie: better bandwidth utilization) ? I'll talk more about your queueing rule in a moment. > 3. Given the importance of time keeping, would it be a good idea to have > another queue for NTP traffic and use the highest priority of 7 for it ? I doubt it would harm anything, but it's probably unnecessary. In general, OpenBSD does the sensible thing. Sometimes improvements can be made through configuration, but it's usually best to clearly define and lay out the problem before deciding on a solution. When tweaking something, have a good understanding of what you're changing and why. Now I'll get more into your configuration. > queue rootq on $ext_if bandwidth 55M max 55M > queue dataq parent rootq bandwidth 55M max 55M flows 1024 \ > qlimit 1024 default As I understand it, that queue statement should be based on upload speed, not download speed. You can shape the traffic others send to you by controlling what you send to them. The nature of TCP congestion control (TCP slow start and friends) is why this approach works. It's like a telephone conversation---if the other person is talking (sending data) and you're silent for a long time (no acknowledgements/ACKs), the other person will ask if you're still there and wait for a response. So upload speed is the relevant metric because it's your response that you have control over. I can't take credit for the telephone analogy, it's from Solene's blog post on the subject[1]. Though I view her configuration itself as a bit of a special case. I've experimented with configurations that complex, and these days I mostly stick to the simple one rule configuration mentioned in pf.conf(5) under QUEUEING. > match out on $ext_if inet proto tcp set queue dataq set prio (5, 6) \ > tag INTERNET I'm aware of the priorities trick you're using[2] and I'd guess that the effect on top of FQ-CoDel + HFSC isn't a lot. But without proper testing, I don't know. Anyway, assigning to a queue and tagging are unnecessary here. The default queue will apply if the packets aren't assigned elsewhere. If the tagging is to make the rule's function clearer to you, then I'd just comment it instead. Also, it's perhaps a bit paranoid but I'd change two things. I'd add 'prio 3' to guard against accidentally overriding nondefault priority values, and I'd change the priority values to '(3, 4)'. The first change is so that it only matches normal priority packets and leaves everything else alone. The second change is so that outbound traffic remains at the default priority and therefore still shares an equal priority with inbound traffic (except for the dataless ACKs and packets with a TOS of lowdelay). As for mixing priorities and bandwidth shaping, I've felt unsure about how exactly they interact for quite a while now. I decided to put that doubt of mine to rest once and for all today, and this is what I could find. It seems that it used to be the case that they couldn't be used together, back in the days of ALTQ[3]. But there's evidence to suggest that they can be used together now[4][5]. I tested it and confirmed as much. > It occurs to me that in my originally proposed configuration, I am not > limiting the traffic with the two priorities to TCP traffic. This is > necessary as this optimization applies only to TCP traffic and I should note > that in Peter Hansteen's book he also does this. Good that you noticed that, but it's unnecessary. pf is smart enough to know what traffic to apply it to. It's good to compare the output of pfctl(8) to know exactly what's changing and how things are getting parsed (`pfctl -s rules`, `pfctl -nvf /etc/pf.conf`). > Does anyone have any feedback or pointers on this updated configuration ? My main suggestion is to test the modifications you're making to get objective results. This is both to build confidence and to make sure that you aren't accidentally screwing up your connection somehow. Usually the reason you'd change things like this is to avoid bufferbloat[6]. I like to use Flent[7] to test bufferbloat, but you're welcome to use web tools as well[8]. Run the test without the queues/priorities first to figure out what the baseline is, then add them back in one at a time and compare. With all this in mind, I'd make some minor changes to what you sent: ``` ext_if = "em0" int_if = "em1" set skip on lo block # Use 90-95% of the expected upload speed for bandwidth shaping. This activates # HFSC + FQ-CoDel. queue outq on $ext_if flows 1024 bandwidth 9M max 9M qlimit 1024 default # Be paranoid and avoid rewriting nondefault packet priority values. Keep # outbound traffic the same priority as inbound traffic (with the exception of # TCP ACKs with no data payload and packets with a TOS of lowdelay---those are # set one above the default of 3). match out on $ext_if prio 3 set prio (3, 4) match out on $ext_if inet from ($int_if:network) nat-to ($ext_if:0) pass in on $int_if pass out on $ext_if ``` Peter's book is good, I've read it as well. Though keep in mind that it's from 2015 and that it's best to compare unofficial resources with official ones. I'd direct you to the example ruleset for a router in the OpenBSD FAQ[9]. It's a good foundation to start from. The only thing I feel mixed about is the scrub rule, but it's probably fine to keep that in. [1]: https://dataswamp.org/~solene/2021-08-30-openbsd-qos-lan.html [2]: https://www.benzedrine.ch/ackpri.html [3]: https://marc.info/?l=openbsd-misc&m=140127924031145&w=2 [4]: https://marc.info/?l=openbsd-tech&m=154019352813682&w=2 [5]: https://marc.info/?l=openbsd-tech&m=153984675725845&w=2 [6]: https://www.bufferbloat.net/projects/ [7]: https://flent.org/ [8]: https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/ [9]: https://www.openbsd.org/faq/pf/example1.html -- https://www.anthes.is/
