Hi,
we have a kernel panic after compiling with witness and invariant
Feb 27 13:49:33 sovapn1 kernel: lock order reversal:
Feb 27 13:49:33 sovapn1 kernel: 1st 0xfffffe000fed78b8 pf_idhash (pf_idhash) @
/usr/src/sys/netpfil/pf/pf.c:1078
Feb 27 13:49:33 sovapn1 kernel: 2nd 0xfffff8001e0474a8 pfsync (pfsync) @
/usr/src/sys/netpfil/pf/if_pfsync.c:1667
Feb 27 13:49:33 sovapn1 kernel: stack backtrace:
Feb 27 13:49:33 sovapn1 kernel: #0 0xffffffff80a9efe0 at witness_debugger+0x70
Feb 27 13:49:33 sovapn1 kernel: #1 0xffffffff80a9eed3 at
witness_checkorder+0xe23
Feb 27 13:49:33 sovapn1 kernel: #2 0xffffffff80a20aad at __mtx_lock_flags+0x9d
Feb 27 13:49:33 sovapn1 kernel: #3 0xffffffff826d9f9d at
pfsync_insert_state+0x7d
Feb 27 13:49:33 sovapn1 kernel: #4 0xffffffff826ea729 at pf_state_insert+0x8b9
Feb 27 13:49:33 sovapn1 kernel: #5 0xffffffff826f444c at pf_test_rule+0x2bbc
Feb 27 13:49:33 sovapn1 kernel: #6 0xffffffff826eec58 at pf_test+0x1598
Feb 27 13:49:33 sovapn1 kernel: #7 0xffffffff826ffb2d at pf_check_in+0x1d
Feb 27 13:49:33 sovapn1 kernel: #8 0xffffffff80b488c4 at pfil_run_hooks+0xa4
Feb 27 13:49:33 sovapn1 kernel: #9 0xffffffff80ba5757 at ip_tryforward+0x157
Feb 27 13:49:33 sovapn1 kernel: #10 0xffffffff80ba7a8a at ip_input+0x38a
Feb 27 13:49:33 sovapn1 kernel: #11 0xffffffff80b47a00 at
netisr_dispatch_src+0x80
Feb 27 13:49:33 sovapn1 kernel: #12 0xffffffff80b309ce at ether_demux+0x15e
Feb 27 13:49:33 sovapn1 kernel: #13 0xffffffff80b317ee at ether_nh_input+0x31e
Feb 27 13:49:33 sovapn1 kernel: #14 0xffffffff80b47a00 at
netisr_dispatch_src+0x80
Feb 27 13:49:33 sovapn1 kernel: #15 0xffffffff80b30d62 at ether_input+0x62
Feb 27 13:49:33 sovapn1 kernel: #16 0xffffffff82671427 at igb_rxeof+0x6e7
Feb 27 13:49:33 sovapn1 kernel: #17 0xffffffff826708fe at igb_msix_que+0xee
we compiled with
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESSoptions WITNESS_SKIPSPIN
options WITNESS_KDB
we went back to the 11.1 release as the relenge branch didn't compile
for us (probably our fault).
Regards
Joe Jones
On 27/02/18 04:40, Kristof Provost wrote:
On 26 Feb 2018, at 17:06, Joe Jones wrote:
Hi Kristof,
we are not updating rules during the test although in production
we will reload the rule set from time to time. We are constantly
adding and removing from tables though, using the DIOCRADDADDRS
and DIOCRDELADDRS ioctl, also DIOCKILLSTATES is being called a
lot. These are all in response to RADIUS events. We tried using
pfctl shell command rather than calling ioctl directly, to check
that it wasn't a problem with how we are calling the ioctl.
That’s interesting.
The panic leads me to suspect something’s wrong with the
kt->pfrkt_ipv4->rt, which would explain why we get the unexpected NULL
result.
My first guess at the cause would be a race condition, where it’s
being modified (through one of the ioctls you do) while the
pfr_pool_get() is walking it.
I don’t immediately see where that’d happen though, because both
DIOCRADDADDRS and DIOCRDELADDRS take the rules lock (and
pfr_pool_get() takes it too).
It might be interesting to run this with these extra asserts (and be
sure to enable INVARIANTS).
|diff --git a/sys/netpfil/pf/pf_table.c b/sys/netpfil/pf/pf_table.c
index 18342a94073..cad9b4ea89f 100644 --- a/sys/netpfil/pf/pf_table.c
+++ b/sys/netpfil/pf/pf_table.c @@ -962,6 +962,8 @@
pfr_unroute_kentry(struct pfr_ktable *kt, struct pfr_kentry *ke)
struct radix_node *rn; struct radix_head *head = NULL; +
PF_RULES_WASSERT(); + if (ke->pfrke_af == AF_INET) head =
&kt->pfrkt_ip4->rh; else if (ke->pfrke_af == AF_INET6) @@ -1855,6
+1859,8 @@ pfr_destroy_ktable(struct pfr_ktable *kt, int flushaddr) {
struct pfr_kentryworkq addrq; + PF_RULES_WASSERT(); + if (flushaddr) {
pfr_enqueue_addrs(kt, &addrq, NULL, 0); pfr_clean_node_mask(kt, &addrq); |
Regards,
Kristof
_______________________________________________
freebsd-pf@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"
