Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
Hello David, On Sun, May 29, 2022 at 12:59:25PM +1000, David Gwynne wrote: > > > > On 24 May 2022, at 17:01, Alexandr Nedvedicky > > wrote: > > > > Hello Hrvoje, > > > > On Mon, May 23, 2022 at 06:34:07PM +0200, Hrvoje Popovski wrote: > >> On 23.5.2022. 10:41, Hrvoje Popovski wrote: > >>> On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: > looks like kind of memory corruption. my bet is use-after-free. > will try to get to it later today. > > does it mean there is no such panic, when we handle IPv4 traffic only? > >>> > >>> Hi, > >>> > >>> yes, it seems that i can't trigger panic with ip4 only traffic, at least > >>> the same way i can with ip6 traffic > >>> > >> > >> All day I'm trying to trigger panic with ip4 and I just can't > > > >interesting. I went through mbuf handling in if_veb.c > >I just could find a single nit, which is most likely unrelated, > >however I think it's still worth to give it a try a diff below. > > > >basically all calls to veb_pf() read as follows: > > m = veb_pf(ifp, ..., m); > >except the one in veb_broadcast(), which readsa as: > > m = veb_pf(ifp, ..., m0); > >I think it is a bug, veb_pf() caller should continue to run > >with packet returned by veb_pf(). > > yes. ok by me. can you fix the same thing in the ipsec handling too? > yes will do. I'm still testing those changes on Hrvoje's playground. there are still few more nits around IPv6 I'd to understand better. It well might be the tools (?pkgtgen?) don't generate a valid IPv6 packets or pf(4) is just too paranoid/picky. thanks and regards sashan
Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
> On 24 May 2022, at 17:01, Alexandr Nedvedicky > wrote: > > Hello Hrvoje, > > On Mon, May 23, 2022 at 06:34:07PM +0200, Hrvoje Popovski wrote: >> On 23.5.2022. 10:41, Hrvoje Popovski wrote: >>> On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: looks like kind of memory corruption. my bet is use-after-free. will try to get to it later today. does it mean there is no such panic, when we handle IPv4 traffic only? >>> >>> Hi, >>> >>> yes, it seems that i can't trigger panic with ip4 only traffic, at least >>> the same way i can with ip6 traffic >>> >> >> All day I'm trying to trigger panic with ip4 and I just can't > >interesting. I went through mbuf handling in if_veb.c >I just could find a single nit, which is most likely unrelated, >however I think it's still worth to give it a try a diff below. > >basically all calls to veb_pf() read as follows: > m = veb_pf(ifp, ..., m); >except the one in veb_broadcast(), which readsa as: > m = veb_pf(ifp, ..., m0); >I think it is a bug, veb_pf() caller should continue to run >with packet returned by veb_pf(). yes. ok by me. can you fix the same thing in the ipsec handling too? > > thanks and > regards > sashan > > 8<---8<---8<--8< > diff --git a/sys/net/if_veb.c b/sys/net/if_veb.c > index 67185fde228..30a002f95a2 100644 > --- a/sys/net/if_veb.c > +++ b/sys/net/if_veb.c > @@ -944,7 +944,7 @@ veb_broadcast(struct veb_softc *sc, struct veb_port *rp, > struct mbuf *m0, >* let pf look at it, but use the veb interface as a proxy. >*/ > if (ISSET(ifp->if_flags, IFF_LINK1) && > - (m = veb_pf(ifp, PF_OUT, m0)) == NULL) > + (m0 = veb_pf(ifp, PF_OUT, m0)) == NULL) > return; > #endif > >
Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
On 24.5.2022. 9:01, Alexandr Nedvedicky wrote:
> interesting. I went through mbuf handling in if_veb.c
> I just could find a single nit, which is most likely unrelated,
> however I think it's still worth to give it a try a diff below.
>
> basically all calls to veb_pf() read as follows:
> m = veb_pf(ifp, ..., m);
> except the one in veb_broadcast(), which readsa as:
> m = veb_pf(ifp, ..., m0);
> I think it is a bug, veb_pf() caller should continue to run
> with packet returned by veb_pf().
>
> thanks and
> regards
> sashan
Hi,
and with this diff i can panic box the same way as before... ip6
forwarding, pf and veb/vport
panic:
r620-1# panuicvm:_ f paoulotl(_0caxcffhfef_iftfeffm8_2ma2gfi13ca_c8h, e
ck :m bu f p
l cp uf r
e0ex1 7 , l i 0s,t 2 ) - > e
mkoedrnieflie: d : i t e m a dd r0 xf f f ff d 8 0 a 42 0 e
5 00 + 2 4 0x 6
a b 22 4 5 9 6 1e e 9 8 5c ! = 0 x 6 ab 2 2 4 5
9pcadge0 a f8 5 c
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
418374 46077 0 0x14000 0x2003 softnet
355064 80120 0 0x14000 0x2002K softnet
*401307 69853 0 0x14000 0x2005 softnet
db_enter() at db_enter+0x10
panic(81f3c6f5) at panic+0xbf
pool_cache_get(82483608) at pool_cache_get+0x25b
pool_get(82483608,2) at pool_get+0x61
m_get(2,1) at m_get+0x3f
m_copym(fd80a3b50900,0,40,2) at m_copym+0xd8
ip6_forward(fd80a3b50900,fd842ce9c708,0) at ip6_forward+0x1cc
ip6_input_if(800022c6b728,800022c6b734,29,0,8074b000) at
ip6_input_if+0x80a
ipv6_input(8074b000,fd80a3b50900) at ipv6_input+0x39
ether_input(8074b000,fd80a3b50900) at ether_input+0x3ad
vport_if_enqueue(8074b000,fd80a3b50900) at vport_if_enqueue+0x19
veb_port_input(80095048,fd80a3b50900,ecf4bbdaf7f8,80747300)
at veb_port_input+0x5b0
ether_input(80095048,fd80a3b50900) at ether_input+0x100
if_input_process(80095048,800022c6b938) at if_input_process+0x6f
end trace frame: 0x800022c6b980, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{5}> show panic
*cpu5: pool_cache_item_magic_check: mbufpl cpu free list modified: item
addr 0x
fd80a420e500+24 0x6ab2245961ee985c!=0x6ab22459cd0af85c
cpu2: uvm_fault(0x822f13a8, 0x17, 0, 2) -> e
ddb{5}>
Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
Hello Hrvoje, On Mon, May 23, 2022 at 06:34:07PM +0200, Hrvoje Popovski wrote: > On 23.5.2022. 10:41, Hrvoje Popovski wrote: > > On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: > >> looks like kind of memory corruption. my bet is use-after-free. > >> will try to get to it later today. > >> > >> does it mean there is no such panic, when we handle IPv4 traffic only? > > > > Hi, > > > > yes, it seems that i can't trigger panic with ip4 only traffic, at least > > the same way i can with ip6 traffic > > > > All day I'm trying to trigger panic with ip4 and I just can't interesting. I went through mbuf handling in if_veb.c I just could find a single nit, which is most likely unrelated, however I think it's still worth to give it a try a diff below. basically all calls to veb_pf() read as follows: m = veb_pf(ifp, ..., m); except the one in veb_broadcast(), which readsa as: m = veb_pf(ifp, ..., m0); I think it is a bug, veb_pf() caller should continue to run with packet returned by veb_pf(). thanks and regards sashan 8<---8<---8<--8< diff --git a/sys/net/if_veb.c b/sys/net/if_veb.c index 67185fde228..30a002f95a2 100644 --- a/sys/net/if_veb.c +++ b/sys/net/if_veb.c @@ -944,7 +944,7 @@ veb_broadcast(struct veb_softc *sc, struct veb_port *rp, struct mbuf *m0, * let pf look at it, but use the veb interface as a proxy. */ if (ISSET(ifp->if_flags, IFF_LINK1) && - (m = veb_pf(ifp, PF_OUT, m0)) == NULL) + (m0 = veb_pf(ifp, PF_OUT, m0)) == NULL) return; #endif
Re: ip6 forwarding with pf and pfsync over veb/vport
On 23.5.2022. 10:41, Hrvoje Popovski wrote: > On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: >> looks like kind of memory corruption. my bet is use-after-free. >> will try to get to it later today. >> >> does it mean there is no such panic, when we handle IPv4 traffic only? > > Hi, > > yes, it seems that i can't trigger panic with ip4 only traffic, at least > the same way i can with ip6 traffic > All day I'm trying to trigger panic with ip4 and I just can't
Re: ip6 forwarding with pf and pfsync over veb/vport
On 23.5.2022. 10:41, Hrvoje Popovski wrote:
> On 23.5.2022. 8:34, Alexandr Nedvedicky wrote:
>> looks like kind of memory corruption. my bet is use-after-free.
>> will try to get to it later today.
>>
>> does it mean there is no such panic, when we handle IPv4 traffic only?
>
> Hi,
>
> yes, it seems that i can't trigger panic with ip4 only traffic, at least
> the same way i can with ip6 traffic
>
Here's another one but this time i've tcpdump outgoing ix interface.
I've tried same stuff with ip4 traffic and couldn't trigger panic.
10:53:59.682513 a192:a168:a100::111.9 > b192:b168:b111::bfbf.9: udp
puvamn_icf:au l t p(o0
oxflf_cfafcffhfe_fi82t2emf_62m6a8gi, c _ ch e c k : m b uf p l c p
uf r
0exe1 l7i, s t m o d if i e d : i t e m a d d r 0 x ff f f f
d8 0 a 37 f d a
0 0+ 1 60 xf f f ff d 8 0a 3 7 fd a f 2! = 0x c
0f1,8 9 2b)ec d f -5>9 b0 0 b
Stopped at db_enter+0x10: popq%rbp
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
32710 85256 0 0x14000 0x2004K softnet
97437 83157 0 0x14000 0x2001 softnet
212200 25091 0 0x14000 0x2003 softnet
510395 50985 0 0x14000 0x2005 softnet
417502 88838 0 0x14000 0x2000 systq
db_enter() at db_enter+0x10
panic(81f34fe0) at panic+0xbf
pool_cache_get(82474c48) at pool_cache_get+0x25b
pool_get(82474c48,2) at pool_get+0x61
m_clget(0,2,802) at m_clget+0xdd
ixgbe_get_buf(800973a0,b2) at ixgbe_get_buf+0xa3
ixgbe_rxfill(800973a0) at ixgbe_rxfill+0xaa
ixgbe_queue_intr(80024d00) at ixgbe_queue_intr+0x4f
intr_handler(800022c89380,80081e00) at intr_handler+0x6e
Xintr_ioapic_edge0_untramp() at Xintr_ioapic_edge0_untramp+0x18f
acpicpu_idle() at acpicpu_idle+0x203
sched_idle(800022412ff0) at sched_idle+0x280
end trace frame: 0x0, count: 3
https://www.openbsd.org/ddb.html describes the minimum info required in
bug reports. Insufficient info makes it difficult to find and fix bugs.
ddb{2}>
ddb{2}> show panic
cpu4: uvm_fault(0x822f6268, 0x17, 0, 2) -> e
*cpu2: pool_cache_item_magic_check: mbufpl cpu free list modified: item
addr 0x
fd80a37fda00+16 0xfd80a37fdaf2!=0xcf189becdf59b00b
ddb{2}>
ddb{2}> show reg
rdi0
rsi 0x14
rbp 0x800022c88ff0
rbx 0xfd842f835c00
rdx 0xc800
rcx0x206
rax 0x8a
r8 0x101010101010101
r9 0
r10 0xe6540fc793a8e615
r11 0x4860824aa7540a0c
r12 0x800022413a60
r130
r140
r15 0x81f34fe0cmd0646_9_tim_udma+0x2acb1
rip 0x817b4d90db_enter+0x10
cs 0x8
rflags 0x206
rsp 0x800022c88ff0
ss 0x10
db_enter+0x10: popq%rbp
ddb{2}> show mbuf
mbuf 0x817b4d90
m_type: -13108 m_flags:
c3cc
m_next: 0x1d3b4c241c334c5d m_nextpkt: 0x117400ae525c
m_data: 0x m_len: 3435973836
m_dat: 0x817b4db0 m_pktdat: 0x817b4e00
ddb{2}> show all locks
Process 85256 (softnet) thread 0x8000e7e0 (32710)
shared rwlock netlock r = 0 (0x822e9990)
shared rwlock softnet r = 0 (0x80031370)
Process 83157 (softnet) thread 0x8000ea80 (97437)
shared rwlock netlock r = 0 (0x822e9990)
shared rwlock softnet r = 0 (0x80031270)
Process 25091 (softnet) thread 0x8000ed20 (212200)
shared rwlock netlock r = 0 (0x822e9990)
shared rwlock softnet r = 0 (0x80031170)
Process 50985 (softnet) thread 0x8000efc0 (510395)
shared rwlock softnet r = 0 (0x80031070)
Process 88838 (systq) thread 0x8000f500 (417502)
shared rwlock systq r = 0 (0x822eaf08)
Process 59744 (softclock) thread 0x8000f7a0 (200127)
exclusive kernel_lock &kernel_lock r = 0 (0x824b03c0)
shared rwlock timeout r = 0 (0x822b2fe8)
ddb{2}> ps
PID TID PPIDUID S FLAGS WAIT COMMAND
81137 105065 65725 76 30x100093 netio tcpdump
65725 227707 17816 76 3 0x1100093 ttyouttcpdump
17816 349982 1 0 30x10008b sigsusp ksh
96985 429538 1 0 30x100098 kqreadcron
95498 144368 28860 95 3 0x1100092 kqreadsmtpd
43714 295842 28860103 3 0x1100092 kqreadsmtpd
80683 116687 28860 95 3 0x1100092 kqreadsmtpd
35950 130878 28860 95 30x100092 kqreadsmtpd
27765 48615 28860 95 3 0x1100092 kqreadsmtpd
55438 323904 28860 95 3 0x1100092 kqreadsmtpd
28860
Re: ip6 forwarding with pf and pfsync over veb/vport
On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: > looks like kind of memory corruption. my bet is use-after-free. > will try to get to it later today. > > does it mean there is no such panic, when we handle IPv4 traffic only? Hi, yes, it seems that i can't trigger panic with ip4 only traffic, at least the same way i can with ip6 traffic
Re: ip6 forwarding with pf and pfsync over veb/vport
Hello Hrvoje,
>
>
> ddb{3}> show panic
> *cpu3: pool_cache_item_magic_check: mbufpl cpu free list modified: item
> addr 0xfd80a41c3f00+24 0xaf551e6f8f90255f!=0xaf551e6f8f35fd5f
> cpu2: uvm_fault(0x823ba618, 0x17, 0, 2) -> e
> ddb{3}>
>
looks like kind of memory corruption. my bet is use-after-free.
will try to get to it later today.
does it mean there is no such panic, when we handle IPv4 traffic only?
thanks for testing
regards
sashan
