Re: multiple nat and rdr rules

[Daniel Melameth]

On Thu, Jul 31, 2008 at 2:35 PM, Jacob Lambert <[EMAIL PROTECTED]> wrote:
> I'm new to pf but am learning quick. I've got one pf box up and running and
> working great. Now I want to try to simplify things a bit.
>
> I have multiple VMs each with their own public IP that need nat and rdr
> rules. For now I've been duplicating the nat and rdr rules for each new
> virtual host.  Currently there's only a few VMs but soon I'll have 30-50 VMs
> each with mostly the same nat and rdr rules.

I know this is not what you're looking for, but I've never been a big
fan of NAT.  With all these public IPs, is ditching NAT an option--and
putting public IPs on these hosts instead?  I promise PF will still
work swimmingly ;) .

Re: PF+ALTQ+PRIQ

[Daniel Melameth]

On Wed, Jul 30, 2008 at 1:23 AM, Rudi Kramer - MWEB <[EMAIL PROTECTED]> wrote:
> I am running FreeBSD 7 and I wanted to play around with ALTQ and PRIQ 
> queuing. My goal was to have TCP ACKs that have no payload having the highest 
> priority and then cod, dns, ssh in their own queues and everything else 
> falling in to the default queue.
>
> Here is the config I came up with:
>
> ##
>
> #Macros
>
> ext_if = "tun0"
> cod_ports = "{28960:29000}"
>
> ##Tables
>
> table  { 192.168.0.0/24 }
>
> ##Options
>
> ##Scrub
>
> scrub in all
>
> ##Queueing
>
> altq on $ext_if priq bandwidth 400Kb queue { q_pri, q_def, q_cod, q_domain, 
> q_ssh }
>
> queue q_pri priority 10
> queue q_cod priority 9
> queue q_domain priority 8
> queue q_ssh priority 7
> queue q_def priority 1 priq(default)
>
> ##Translation
> nat on $ext_if from  to any -> ($ext_if)
>
> ##Filter Rules
>
> #default to deny
> block in log all
>
> #allow loopback
> pass quick on lo0 all
>
> #Setup PRIQ Rules
> pass out on $ext_if proto tcp from ($ext_if) to any queue (q_pri, q_def) pass 
> in  on $ext_if proto tcp from any to ($ext_if) queue (q_pri, q_def)

I believe the ordering of the queues here is wrong.  I believe you'll
want this instead:

pass out on $ext_if proto tcp from ($ext_if) to any queue (q_def, q_pri)

And you'll want the same for the identical pass in rule, but since you
are really not allowing connections to come in (state is being kept on
connections that are initiated outbound so pf is tracking the related
inbound packets too) this rule is not needed.

> pass out quick on $ext_if proto udp from ($ext_if) to any port $cod_ports 
> queue q_cod pass in  quick on $ext_if proto udp from any to ($ext_if) port 
> $cod_ports queue q_cod
>
> pass out quick on $ext_if proto udp from ($ext_if) to any port domain queue 
> q_domain pass in  quick on $ext_if proto udp from any to ($ext_if) port 
> domain queue q_domain
>
> pass out quick on $ext_if proto tcp from ($ext_if) to any port ssh queue 
> q_ssh pass in  quick  on $ext_if proto tcp from any to ($ext_if) port ssh 
> queue q_ssh
>
> #allow from fw to ext
> pass out quick log on $ext_if proto tcp all pass out quick log on $ext_if 
> proto { udp, icmp } all

Because your first rule for tcp, the one I commented on, does not have
quick applied, it is likely this tcp rule is catching all the tcp
traffic instead and queueing is not being applied the way you want it
to.  Unless you know what you are doing, I do not recommend using
regular and quick rules in the same ruleset and recommend you focus on
the proper ordering of your rules instead.  For now, you might want to
consider omitting these two rules.  However, in seeing that you are
using this rule in an attempt to treat the firewall's own traffic
differently from NATted traffic, I'm not sure this is possible without
tagging of some sort--which you'll probably need to read up on.

> #allow from internal network out
> pass quick log on $int_if proto tcp from  to any pass quick log on 
> $int_if proto {udp, icmp } from  to any
>
> #
>
> As far as I can see it is working but I was hoping to get some input from the 
> list.

Use pfctl -vvs q to help verify the proper operation of your queues.

Hope this helps.

Re: altq priq Anomaly (Solved)

[Daniel Melameth]

On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote:
> On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote:
> > On 2007/07/20 15:20, Daniel Melameth wrote:
> > > then go back to the broken behavior sometime later.  A reboot of the box 
> > > or
> > > removing altq is the only way to resolve the issue, temporarily.  I've 
> > > tried
> > > both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher
> > > HZ value and using different hardware and different Internet connections,
> > > but the issue persists.
> >
> > Try a snapshot, i386 moved to the new timecounter code after 4.1.
> >
> > Though I note there is also an XXX about variable cpu frequency (in
> > sys/altq/altq_subr.c) which might affect you if you adjust cpu speed
> > (manually or by apmd).
>
> Thanks for taking the time to reply.  I can't readily do a snapshot
> now, but since I am using apmd, I'll try this avenue first and see
> what happens.

Thanks to Stuart's review of the altq code, I have, finally, resolved
this issue which has affected me for years.  Disabling apmd did not
resolve the issue, but disabling the Dynamically Switchable setting
for CPU Frequency in the BIOS appears to have addressed this problem.
I have been using the system for a couple of months now and altq has
been very solid.

Re: altq priq Anomaly

[Daniel Melameth]

On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote:
> On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote:
> > On 2007/07/20 15:20, Daniel Melameth wrote:
> > > then go back to the broken behavior sometime later.  A reboot of the box 
> > > or
> > > removing altq is the only way to resolve the issue, temporarily.  I've 
> > > tried
> > > both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher
> > > HZ value and using different hardware and different Internet connections,
> > > but the issue persists.
> >
> > Try a snapshot, i386 moved to the new timecounter code after 4.1.
> >
> > Though I note there is also an XXX about variable cpu frequency (in
> > sys/altq/altq_subr.c) which might affect you if you adjust cpu speed
> > (manually or by apmd).
> >
> > > Full detail on the issue is available from my gnats post at
> > > # Attempt to account for overhead of RFC 1483 bridging, ATM, PPPoE and TCP
> >
> > For this you might like to try a diff of mine and report back
> > (http://spacehopper.org/openbsd/altq_tbradapt.diff).
> > It won't help the other problem though.
>
> Thanks for taking the time to reply.  I can't readily do a snapshot
> now, but since I am using apmd, I'll try this avenue first and see
> what happens.  I also went ahead and incorporated your diffs into my
> currently running 4.1-stable and it appears to be working quite well,
> but I'm not certain if I'm piloting these changes the correct way--so
> if there's an ideal way you'd recommend piloting this, let me know.
> Regardless, I'll monitor the result of these diffs this week and give
> you a heads up at the end of the week.

Stuart,

I did some simple piloting of your diffs and they work quite nicely.
The details:

Without altq, I get a fairly consistent roundtrip latency of 58ms to a
known good host when there is no congestion--and when the upstream is
saturated, latency to the same host goes up to ~220ms.  If I use altq
without your diffs and specify my maximum upload bandwidth of 896Kb,
as expected the overhead of PPPoE tramples this and I still get ~220ms
under saturation even with a high priority queue.  In order to get
decency latency in this case, I have to artificially estimate a
bandwidth of ~716Kb--and this provides for a high priority queue
latency of ~68ms while saturated, which is pretty good.

With your diffs though, not only do I get ~68ms for a high priority
queue while using a bandwidth of 896Kb on a saturated connection, but
my maximum upload speed is, consistently, a bit higher as well when
compared to using a bandwidth of 716Kb without your diffs.  I
continued to pilot your diffs further by seeding and downloading
multiple torrents simultaneously--to more heavily expose small TCP
ACKs to the stream--and I continued to get these consistently good
results.

I have been using your diffs for almost a week now with nary an issue.
 So, my next question is:

HOW DO WE GET YOUR GREAT DIFFS INTO -CURRRENT OR A SNAPSHOT?

Re: altq priq Anomaly

[Daniel Melameth]

On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote:

On 2007/07/20 15:20, Daniel Melameth wrote:
> then go back to the broken behavior sometime later.  A reboot of the box or
> removing altq is the only way to resolve the issue, temporarily.  I've tried
> both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher
> HZ value and using different hardware and different Internet connections,
> but the issue persists.

Try a snapshot, i386 moved to the new timecounter code after 4.1.

Though I note there is also an XXX about variable cpu frequency (in
sys/altq/altq_subr.c) which might affect you if you adjust cpu speed
(manually or by apmd).

> Full detail on the issue is available from my gnats post at
> # Attempt to account for overhead of RFC 1483 bridging, ATM, PPPoE and TCP

For this you might like to try a diff of mine and report back
(http://spacehopper.org/openbsd/altq_tbradapt.diff).
It won't help the other problem though.


Thanks for taking the time to reply.  I can't readily do a snapshot
now, but since I am using apmd, I'll try this avenue first and see
what happens.  I also went ahead and incorporated your diffs into my
currently running 4.1-stable and it appears to be working quite well,
but I'm not certain if I'm piloting these changes the correct way--so
if there's an ideal way you'd recommend piloting this, let me know.
Regardless, I'll monitor the result of these diffs this week and give
you a heads up at the end of the week.

The only breakage I've noticed so far appears to be related to pfstat
which reports the following (and I assume this is related to the new
headers, but I haven't looked into it yet):

ioctl: DIOCGETALTQS: Permission denied
pf_query: query_queues() failed

Thanks again Stuart.

Re: altq priq Anomaly

[Daniel Melameth]

A recompile of pfstat has addressed this.

On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote:

Thanks for taking the time to reply.  I can't readily do a snapshot
now, but since I am using apmd, I'll try this avenue first and see
what happens.  I also went ahead and incorporated your diffs into my
currently running 4.1-stable and it appears to be working quite well,
but I'm not certain if I'm piloting these changes the correct way--so
if there's an ideal way you'd recommend piloting this, let me know.
Regardless, I'll monitor the result of these diffs this week and give
you a heads up at the end of the week.

The only breakage I've noticed so far appears to be related to pfstat
which reports the following (and I assume this is related to the new
headers, but I haven't looked into it yet):

ioctl: DIOCGETALTQS: Permission denied
pf_query: query_queues() failed

altq priq Anomaly

[Daniel Melameth]

Because of this ever nagging issue, I thought I'd try posting here again,
two years since my last post.  For years now, since OpenBSD 3.5, I have been
unable to get altq to limit bandwidth properly.  In a nutshell, if I give
altq an example bandwidth value of 1024Kb, my bandwidth will be incorrectly
capped to about 512Kb or less.  This issue doesn't occur immediately-it only
happens after the box has an uptime of like a few weeks or so and then,
perhaps after an uptime of over two months, it'll work normally again and
then go back to the broken behavior sometime later.  A reboot of the box or
removing altq is the only way to resolve the issue, temporarily.  I've tried
both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher
HZ value and using different hardware and different Internet connections,
but the issue persists.

 

Full detail on the issue is available from my gnats post at
http://cvs.openbsd.org/cgi-bin/query-pr-wrapper?full=yes
 &textonly=yes&numbers=4312.  The relevant portion of pf.conf:

 

altq on $external_if priq bandwidth 1522Kb queue \

{ interac, tcp_ack, default, torrent }

queue interac priority 14

queue tcp_ack priority 13

queue default priority 8 priq ( default )

queue torrent priority 2 priq ( red )

 

dmesg from a piece of hardware I'm experiencing this issue with.  Any
thoughts/guidance is appreciated.

 

OpenBSD 4.1 (GENERIC) #1435: Sat Mar 10 19:07:45 MST 2007

[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC

cpu0: Intel(R) Pentium(R) M processor 1600MHz ("GenuineIntel" 686-class)
1.60 GHz

cpu0:
FPU,V86,DE,PSE,TSC,MSR,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,
FXSR,SSE,SSE2,TM,SBF,EST,TM2

real mem  = 536244224 (523676K)

avail mem = 481591296 (470304K)

using 4278 buffers containing 26935296 bytes (26304K) of memory

mainbus0 (root)

bios0 at mainbus0: AT/286+ BIOS, date 06/23/05, BIOS32 rev. 0 @ 0xf,
SMBIOS rev. 2.3 @ 0xfa1ee (31 entries)

bios0: Hewlett-Packard HP Compaq nc6000 (DQ880A#ABA)

apm0 at bios0: Power Management spec V1.2 (BIOS managing devices)

apm0: AC on, battery charge unknown

apm0: flags 130102 dobusy 0 doidle 1

pcibios0 at bios0: rev 2.1 @ 0xf/0x2000

pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf0840/160 (8 entries)

pcibios0: bad IRQ table checksum

pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf6360/160 (8 entries)

pcibios0: PCI Exclusive IRQs: 5 10 11

pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801DBM LPC" rev 0x00)

pcibios0: PCI bus #5 is the last bus

bios0: ROM list: 0xc/0x1

acpi at mainbus0 not configured

cpu0 at mainbus0

cpu0: Enhanced SpeedStep 1600 MHz (1484 mV): speeds: 1600, 1400, 1200, 1000,
800, 600 MHz

pci0 at mainbus0 bus 0: configuration mode 1 (no bios)

pchb0 at pci0 dev 0 function 0 "Intel 82855PE Hub" rev 0x03

ppb0 at pci0 dev 1 function 0 "Intel 82855PE AGP" rev 0x03

pci1 at ppb0 bus 1

vga1 at pci1 dev 0 function 0 "ATI Radeon Mobility M10 NP" rev 0x00

wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)

wsdisplay0: screen 1-5 added (80x25, vt100 emulation)

uhci0 at pci0 dev 29 function 0 "Intel 82801DB USB" rev 0x03: irq 10

usb0 at uhci0: USB revision 1.0

uhub0 at usb0

uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1

uhub0: 2 ports with 2 removable, self powered

uhci1 at pci0 dev 29 function 1 "Intel 82801DB USB" rev 0x03: irq 10

usb1 at uhci1: USB revision 1.0

uhub1 at usb1

uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1

uhub1: 2 ports with 2 removable, self powered

uhci2 at pci0 dev 29 function 2 "Intel 82801DB USB" rev 0x03: irq 10

usb2 at uhci2: USB revision 1.0

uhub2 at usb2

uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1

uhub2: 2 ports with 2 removable, self powered

ehci0 at pci0 dev 29 function 7 "Intel 82801DB USB" rev 0x03: irq 10

usb3 at ehci0: USB revision 2.0

uhub3 at usb3

uhub3: Intel EHCI root hub, rev 2.00/1.00, addr 1

uhub3: 6 ports with 6 removable, self powered

ppb1 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0x83

pci2 at ppb1 bus 2

ath0 at pci2 dev 4 function 0 "Atheros AR5212" rev 0x01: irq 11

ath0: AR5213 5.6 phy 4.1 rf5111 1.7 rf2111 2.3, WOR0W, address
00:0b:cd:5a:28:bb

cbb0 at pci2 dev 6 function 0 "O2 Micro OZ711E0 CardBus" rev 0x00: irq 10

cbb1 at pci2 dev 6 function 1 "O2 Micro OZ711E0 CardBus" rev 0x00: irq 10

"O2 Micro OZ711Mx CardBus" rev 0x00 at pci2 dev 6 function 2 not configured

cbb2 at pci2 dev 6 function 3 "O2 Micro OZ711E0 CardBus" rev 0x00: irq 10

bge0 at pci2 dev 14 function 0 "Broadcom BCM5705M Alt" rev 0x03, BCM5705 A3
(0x3003): irq 11, address 00:08:02:de:d3:2d

brgphy0 at bge0 phy 1: BCM5705 10/100/1000baseT PHY, rev. 2

cardslot0 at cbb0 slot 0 flags 0

cardbus0 at cardslot0: bus 3 device 0 cacheline 0x0, lattimer 0x20

pcmcia0 at cardslot0

cardslot1 at cbb1 slot 1 flags 0

cardbus1 at cardslot1: bus 4 device 0 cacheline 0x0, lattimer 0x20

pcmcia1 at cardslot1

"Small" Networks and urpf-failed

[Daniel Melameth]

I definitely see the value of urpf-failed-especially in bgp or similar
environments.  The question I have is, is this an "expensive" way to prevent
spoofing?  In other works, does it use significantly more CPU time when
compared to something like antispoof?  Also, does urpf-failed "flag"
incoming packets that are sourced with the same IP as the firewall like
antispoof does?

 

Thanks.