I'll be sure to run those commands in ddb the next time this happens.
I have some dynamic activity. One script I have adds and deletes my home IP
from a table when my home IP changes. Though, this didn't happen in this
case.
It does read from that table every 5 minutes. I also have pf doing rate
limiting on some ports and adding to a bruteforce table. That table then
goes
through a queue with a low bandwidth. I also have these cron jobs running
50 9 * * * -n smtpctl spf walk <
/etc/mail/nospamd_domains.txt >/etc/mail/nospamd
0 10 * * * -n pfctl -T replace -t nospamd
-f /etc/mail/nospamd
The time of the crash doesn't coincide with any of those times, however.
Courtney
On 4/22/25 4:23 AM, Alexander Bluhm wrote:
On Thu, Apr 10, 2025 at 01:15:01PM -0700, Courtney wrote:
I've had this crash happen a couple times, and I remembered this time to
get the trace:
kernel: protection fault trap, code=0
Stopped at pf_counters_inc+0x190: addq %rax,0x408(%rcx,%r13,8)
ddb{0}> trace
pf_counters_inc(0,ffff80001c976df0,fffffd802b2269e0,ffff800000938018,0)
at pf_counters_inc+0x190
pf_test(18,1,ffff80000004d058,ffff80001c9770c8) at pf_test+0x822
ip6_input_if(ffff80001c9770c8,ffff80001c9770d4,29,0,ffff80000004d058) at
ip6_input_if+0x1ce
ipv6_input(ffff80000004d058,fffffd80186c5000) at ipv6_input+0x3d
ether_input(ffff80000004d058,fffffd80186c5000) at ether_input+0x3df
ifiq_process(ffff80000004d468) at ifiq_process+0x90
taskq_thread(ffff800000030000) at taskq_thread+0x129
end trace frame: 0x0, count: -8
The crash happens here.
0000000000016010 <pf_counters_inc>:
/usr/src/sys/net/pf.c:7505
1618c: 48 8b 43 08 mov 0x8(%rbx),%rax
16190: 4a ff 84 e8 f8 03 00 incq 0x3f8(%rax,%r13,8)
16197: 00
/usr/src/sys/net/pf.c:7506
16198: 49 8b 46 10 mov 0x10(%r14),%rax
1619c: 48 8b 4b 08 mov 0x8(%rbx),%rcx
* 161a0: 4a 01 84 e9 08 04 00 add %rax,0x408(%rcx,%r13,8)
161a7: 00
/usr/src/sys/net/pf.c:7508
0000000000013700 <pf_test>:
/usr/src/sys/net/pf.c:7932
13f00: 45 0f b7 f7 movzwl %r15w,%r14d
13f04: 48 8b 4d 98 mov 0xffffffffffffff98(%rbp),%rcx
13f08: 4c 8b 45 80 mov 0xffffffffffffff80(%rbp),%r8
13f0c: 48 8d b5 a0 fe ff ff lea 0xfffffffffffffea0(%rbp),%rsi
13f13: 44 89 f7 mov %r14d,%edi
13f16: 48 8b 5d 90 mov 0xffffffffffffff90(%rbp),%rbx
13f1a: 48 89 da mov %rbx,%rdx
13f1d: e8 00 00 00 00 callq 13f22 <pf_test+0x822>
/usr/src/sys/net/pf.c:7934
* 13f22: 48 c7 44 24 f8 00 00 movq $0x0,0xfffffffffffffff8(%rsp)
13f29: 00 00
13f2b: 41 83 fe 0a cmp $0xa,%r14d
13f2f: 0f 8e 9a 00 00 00 jle 13fcf <pf_test+0x8cf>
13f35: 41 83 fe 0b cmp $0xb,%r14d
13f39: 0f 84 a1 01 00 00 je 140e0 <pf_test+0x9e0>
13f3f: 41 83 fe 0d cmp $0xd,%r14d
13f43: 0f 84 b0 00 00 00 je 13ff9 <pf_test+0x8f9>
13f49: 41 83 fe 0f cmp $0xf,%r14d
13f4d: 0f 85 1e 01 00 00 jne 14071 <pf_test+0x971>
13f53: 48 8d bd a0 fe ff ff lea 0xfffffffffffffea0(%rbp),%rdi
/usr/src/sys/net/pf.c:7959
net/pf.c
pf_counters_inc()
7504 SLIST_FOREACH(ri, &st->match_rules, entry) {
7505 ri->r->packets[dirndx]++;
* 7506 ri->r->bytes[dirndx] += pd->tot_len;
7507
7508 if (ri->r->src.addr.type ==
PF_ADDR_TABLE)
7509
pfr_update_stats(ri->r->src.addr.p.tbl,
pf_test()
7930 #endif /* NPFLOG > 0 */
7931
* 7932 pf_counters_inc(action, &pd, st, r, a);
7933
7934 switch (action) {
7935 case PF_SYNPROXY_DROP:
7936 m_freem(pd.m);
7937 /* FALLTHROUGH */
The strange thing is that line 7505 succeeds, but 7506 crashes
although it nearly the same memory.
I don't have a ton of insights. It is my mail server.
Could you please type "show register" into ddb when it happens next
time? Then I might see whether pf_rule_item, pf_rule, or dirndx
is invalid.
You could also analyse the content of pf_rule_item in ddb.
For that please do
ddb> show register
...
rbx 0xffff80002a278f98
rcx 0xffff800000139e00
rbx is the address of pf_rule_item, rcx is pf_rule. You will have
different hex values.
Then type
ddb> show struct pf_rule_item `hex value behind rbx from show register`
ddb> show struct pf_rule `hex value behind rcx from show register`
Is there anything dynamic on your machine regarding pf? Loading
new rule sets with pfctl, configuring new interfaces, new addresses
with autoconf, daemons entering rules in anchors like ftp-proxy(8),
dynamic pf tables from spamd(8), ...
bluhm
