Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
Hello David, On Sun, May 29, 2022 at 12:59:25PM +1000, David Gwynne wrote: > > > > On 24 May 2022, at 17:01, Alexandr Nedvedicky > > wrote: > > > > Hello Hrvoje, > > > > On Mon, May 23, 2022 at 06:34:07PM +0200, Hrvoje Popovski wrote: > >> On 23.5.2022. 10:41, Hrvoje Popovski wrote: > >>> On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: > looks like kind of memory corruption. my bet is use-after-free. > will try to get to it later today. > > does it mean there is no such panic, when we handle IPv4 traffic only? > >>> > >>> Hi, > >>> > >>> yes, it seems that i can't trigger panic with ip4 only traffic, at least > >>> the same way i can with ip6 traffic > >>> > >> > >> All day I'm trying to trigger panic with ip4 and I just can't > > > >interesting. I went through mbuf handling in if_veb.c > >I just could find a single nit, which is most likely unrelated, > >however I think it's still worth to give it a try a diff below. > > > >basically all calls to veb_pf() read as follows: > > m = veb_pf(ifp, ..., m); > >except the one in veb_broadcast(), which readsa as: > > m = veb_pf(ifp, ..., m0); > >I think it is a bug, veb_pf() caller should continue to run > >with packet returned by veb_pf(). > > yes. ok by me. can you fix the same thing in the ipsec handling too? > yes will do. I'm still testing those changes on Hrvoje's playground. there are still few more nits around IPv6 I'd to understand better. It well might be the tools (?pkgtgen?) don't generate a valid IPv6 packets or pf(4) is just too paranoid/picky. thanks and regards sashan
Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
> On 24 May 2022, at 17:01, Alexandr Nedvedicky > wrote: > > Hello Hrvoje, > > On Mon, May 23, 2022 at 06:34:07PM +0200, Hrvoje Popovski wrote: >> On 23.5.2022. 10:41, Hrvoje Popovski wrote: >>> On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: looks like kind of memory corruption. my bet is use-after-free. will try to get to it later today. does it mean there is no such panic, when we handle IPv4 traffic only? >>> >>> Hi, >>> >>> yes, it seems that i can't trigger panic with ip4 only traffic, at least >>> the same way i can with ip6 traffic >>> >> >> All day I'm trying to trigger panic with ip4 and I just can't > >interesting. I went through mbuf handling in if_veb.c >I just could find a single nit, which is most likely unrelated, >however I think it's still worth to give it a try a diff below. > >basically all calls to veb_pf() read as follows: > m = veb_pf(ifp, ..., m); >except the one in veb_broadcast(), which readsa as: > m = veb_pf(ifp, ..., m0); >I think it is a bug, veb_pf() caller should continue to run >with packet returned by veb_pf(). yes. ok by me. can you fix the same thing in the ipsec handling too? > > thanks and > regards > sashan > > 8<---8<---8<--8< > diff --git a/sys/net/if_veb.c b/sys/net/if_veb.c > index 67185fde228..30a002f95a2 100644 > --- a/sys/net/if_veb.c > +++ b/sys/net/if_veb.c > @@ -944,7 +944,7 @@ veb_broadcast(struct veb_softc *sc, struct veb_port *rp, > struct mbuf *m0, >* let pf look at it, but use the veb interface as a proxy. >*/ > if (ISSET(ifp->if_flags, IFF_LINK1) && > - (m = veb_pf(ifp, PF_OUT, m0)) == NULL) > + (m0 = veb_pf(ifp, PF_OUT, m0)) == NULL) > return; > #endif > >
Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
On 24.5.2022. 9:01, Alexandr Nedvedicky wrote: > interesting. I went through mbuf handling in if_veb.c > I just could find a single nit, which is most likely unrelated, > however I think it's still worth to give it a try a diff below. > > basically all calls to veb_pf() read as follows: > m = veb_pf(ifp, ..., m); > except the one in veb_broadcast(), which readsa as: > m = veb_pf(ifp, ..., m0); > I think it is a bug, veb_pf() caller should continue to run > with packet returned by veb_pf(). > > thanks and > regards > sashan Hi, and with this diff i can panic box the same way as before... ip6 forwarding, pf and veb/vport panic: r620-1# panuicvm:_ f paoulotl(_0caxcffhfef_iftfeffm8_2ma2gfi13ca_c8h, e ck :m bu f p l cp uf r e0ex1 7 , l i 0s,t 2 ) - > e mkoedrnieflie: d : i t e m a dd r0 xf f f ff d 8 0 a 42 0 e 5 00 + 2 4 0x 6 a b 22 4 5 9 6 1e e 9 8 5c ! = 0 x 6 ab 2 2 4 5 9pcadge0 a f8 5 c Stopped at db_enter+0x10: popq%rbp TIDPIDUID PRFLAGS PFLAGS CPU COMMAND 418374 46077 0 0x14000 0x2003 softnet 355064 80120 0 0x14000 0x2002K softnet *401307 69853 0 0x14000 0x2005 softnet db_enter() at db_enter+0x10 panic(81f3c6f5) at panic+0xbf pool_cache_get(82483608) at pool_cache_get+0x25b pool_get(82483608,2) at pool_get+0x61 m_get(2,1) at m_get+0x3f m_copym(fd80a3b50900,0,40,2) at m_copym+0xd8 ip6_forward(fd80a3b50900,fd842ce9c708,0) at ip6_forward+0x1cc ip6_input_if(800022c6b728,800022c6b734,29,0,8074b000) at ip6_input_if+0x80a ipv6_input(8074b000,fd80a3b50900) at ipv6_input+0x39 ether_input(8074b000,fd80a3b50900) at ether_input+0x3ad vport_if_enqueue(8074b000,fd80a3b50900) at vport_if_enqueue+0x19 veb_port_input(80095048,fd80a3b50900,ecf4bbdaf7f8,80747300) at veb_port_input+0x5b0 ether_input(80095048,fd80a3b50900) at ether_input+0x100 if_input_process(80095048,800022c6b938) at if_input_process+0x6f end trace frame: 0x800022c6b980, count: 0 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{5}> show panic *cpu5: pool_cache_item_magic_check: mbufpl cpu free list modified: item addr 0x fd80a420e500+24 0x6ab2245961ee985c!=0x6ab22459cd0af85c cpu2: uvm_fault(0x822f13a8, 0x17, 0, 2) -> e ddb{5}>
Re: [External] : Re: ip6 forwarding with pf and pfsync over veb/vport
Hello Hrvoje, On Mon, May 23, 2022 at 06:34:07PM +0200, Hrvoje Popovski wrote: > On 23.5.2022. 10:41, Hrvoje Popovski wrote: > > On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: > >> looks like kind of memory corruption. my bet is use-after-free. > >> will try to get to it later today. > >> > >> does it mean there is no such panic, when we handle IPv4 traffic only? > > > > Hi, > > > > yes, it seems that i can't trigger panic with ip4 only traffic, at least > > the same way i can with ip6 traffic > > > > All day I'm trying to trigger panic with ip4 and I just can't interesting. I went through mbuf handling in if_veb.c I just could find a single nit, which is most likely unrelated, however I think it's still worth to give it a try a diff below. basically all calls to veb_pf() read as follows: m = veb_pf(ifp, ..., m); except the one in veb_broadcast(), which readsa as: m = veb_pf(ifp, ..., m0); I think it is a bug, veb_pf() caller should continue to run with packet returned by veb_pf(). thanks and regards sashan 8<---8<---8<--8< diff --git a/sys/net/if_veb.c b/sys/net/if_veb.c index 67185fde228..30a002f95a2 100644 --- a/sys/net/if_veb.c +++ b/sys/net/if_veb.c @@ -944,7 +944,7 @@ veb_broadcast(struct veb_softc *sc, struct veb_port *rp, struct mbuf *m0, * let pf look at it, but use the veb interface as a proxy. */ if (ISSET(ifp->if_flags, IFF_LINK1) && - (m = veb_pf(ifp, PF_OUT, m0)) == NULL) + (m0 = veb_pf(ifp, PF_OUT, m0)) == NULL) return; #endif
Re: ip6 forwarding with pf and pfsync over veb/vport
On 23.5.2022. 10:41, Hrvoje Popovski wrote: > On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: >> looks like kind of memory corruption. my bet is use-after-free. >> will try to get to it later today. >> >> does it mean there is no such panic, when we handle IPv4 traffic only? > > Hi, > > yes, it seems that i can't trigger panic with ip4 only traffic, at least > the same way i can with ip6 traffic > All day I'm trying to trigger panic with ip4 and I just can't
Re: ip6 forwarding with pf and pfsync over veb/vport
On 23.5.2022. 10:41, Hrvoje Popovski wrote: > On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: >> looks like kind of memory corruption. my bet is use-after-free. >> will try to get to it later today. >> >> does it mean there is no such panic, when we handle IPv4 traffic only? > > Hi, > > yes, it seems that i can't trigger panic with ip4 only traffic, at least > the same way i can with ip6 traffic > Here's another one but this time i've tcpdump outgoing ix interface. I've tried same stuff with ip4 traffic and couldn't trigger panic. 10:53:59.682513 a192:a168:a100::111.9 > b192:b168:b111::bfbf.9: udp puvamn_icf:au l t p(o0 oxflf_cfafcffhfe_fi82t2emf_62m6a8gi, c _ ch e c k : m b uf p l c p uf r 0exe1 l7i, s t m o d if i e d : i t e m a d d r 0 x ff f f f d8 0 a 37 f d a 0 0+ 1 60 xf f f ff d 8 0a 3 7 fd a f 2! = 0x c 0f1,8 9 2b)ec d f -5>9 b0 0 b Stopped at db_enter+0x10: popq%rbp TIDPIDUID PRFLAGS PFLAGS CPU COMMAND 32710 85256 0 0x14000 0x2004K softnet 97437 83157 0 0x14000 0x2001 softnet 212200 25091 0 0x14000 0x2003 softnet 510395 50985 0 0x14000 0x2005 softnet 417502 88838 0 0x14000 0x2000 systq db_enter() at db_enter+0x10 panic(81f34fe0) at panic+0xbf pool_cache_get(82474c48) at pool_cache_get+0x25b pool_get(82474c48,2) at pool_get+0x61 m_clget(0,2,802) at m_clget+0xdd ixgbe_get_buf(800973a0,b2) at ixgbe_get_buf+0xa3 ixgbe_rxfill(800973a0) at ixgbe_rxfill+0xaa ixgbe_queue_intr(80024d00) at ixgbe_queue_intr+0x4f intr_handler(800022c89380,80081e00) at intr_handler+0x6e Xintr_ioapic_edge0_untramp() at Xintr_ioapic_edge0_untramp+0x18f acpicpu_idle() at acpicpu_idle+0x203 sched_idle(800022412ff0) at sched_idle+0x280 end trace frame: 0x0, count: 3 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{2}> ddb{2}> show panic cpu4: uvm_fault(0x822f6268, 0x17, 0, 2) -> e *cpu2: pool_cache_item_magic_check: mbufpl cpu free list modified: item addr 0x fd80a37fda00+16 0xfd80a37fdaf2!=0xcf189becdf59b00b ddb{2}> ddb{2}> show reg rdi0 rsi 0x14 rbp 0x800022c88ff0 rbx 0xfd842f835c00 rdx 0xc800 rcx0x206 rax 0x8a r8 0x101010101010101 r9 0 r10 0xe6540fc793a8e615 r11 0x4860824aa7540a0c r12 0x800022413a60 r130 r140 r15 0x81f34fe0cmd0646_9_tim_udma+0x2acb1 rip 0x817b4d90db_enter+0x10 cs 0x8 rflags 0x206 rsp 0x800022c88ff0 ss 0x10 db_enter+0x10: popq%rbp ddb{2}> show mbuf mbuf 0x817b4d90 m_type: -13108 m_flags: c3cc m_next: 0x1d3b4c241c334c5d m_nextpkt: 0x117400ae525c m_data: 0x m_len: 3435973836 m_dat: 0x817b4db0 m_pktdat: 0x817b4e00 ddb{2}> show all locks Process 85256 (softnet) thread 0x8000e7e0 (32710) shared rwlock netlock r = 0 (0x822e9990) shared rwlock softnet r = 0 (0x80031370) Process 83157 (softnet) thread 0x8000ea80 (97437) shared rwlock netlock r = 0 (0x822e9990) shared rwlock softnet r = 0 (0x80031270) Process 25091 (softnet) thread 0x8000ed20 (212200) shared rwlock netlock r = 0 (0x822e9990) shared rwlock softnet r = 0 (0x80031170) Process 50985 (softnet) thread 0x8000efc0 (510395) shared rwlock softnet r = 0 (0x80031070) Process 88838 (systq) thread 0x8000f500 (417502) shared rwlock systq r = 0 (0x822eaf08) Process 59744 (softclock) thread 0x8000f7a0 (200127) exclusive kernel_lock &kernel_lock r = 0 (0x824b03c0) shared rwlock timeout r = 0 (0x822b2fe8) ddb{2}> ps PID TID PPIDUID S FLAGS WAIT COMMAND 81137 105065 65725 76 30x100093 netio tcpdump 65725 227707 17816 76 3 0x1100093 ttyouttcpdump 17816 349982 1 0 30x10008b sigsusp ksh 96985 429538 1 0 30x100098 kqreadcron 95498 144368 28860 95 3 0x1100092 kqreadsmtpd 43714 295842 28860103 3 0x1100092 kqreadsmtpd 80683 116687 28860 95 3 0x1100092 kqreadsmtpd 35950 130878 28860 95 30x100092 kqreadsmtpd 27765 48615 28860 95 3 0x1100092 kqreadsmtpd 55438 323904 28860 95 3 0x1100092 kqreadsmtpd 28860
Re: ip6 forwarding with pf and pfsync over veb/vport
On 23.5.2022. 8:34, Alexandr Nedvedicky wrote: > looks like kind of memory corruption. my bet is use-after-free. > will try to get to it later today. > > does it mean there is no such panic, when we handle IPv4 traffic only? Hi, yes, it seems that i can't trigger panic with ip4 only traffic, at least the same way i can with ip6 traffic
Re: ip6 forwarding with pf and pfsync over veb/vport
Hello Hrvoje, > > > ddb{3}> show panic > *cpu3: pool_cache_item_magic_check: mbufpl cpu free list modified: item > addr 0xfd80a41c3f00+24 0xaf551e6f8f90255f!=0xaf551e6f8f35fd5f > cpu2: uvm_fault(0x823ba618, 0x17, 0, 2) -> e > ddb{3}> > looks like kind of memory corruption. my bet is use-after-free. will try to get to it later today. does it mean there is no such panic, when we handle IPv4 traffic only? thanks for testing regards sashan
ip6 forwarding with pf and pfsync over veb/vport
Hi all, I can reproduce panic when sending ip6 traffic over vport and destroying pfsync interface. It is reproducible with veb and vport but i couldn't trigger panic when forwarding ip6 over physical interfaces. I've compiled kernel with source fetched half an hour ago just to enable WITNESS. r620-1# ifconfig pfsync0 destroy panicu:v m_ f a u lt ( 0 x f ff f f ff f 8 2 3 ba 6 1 8 , 0 x 17 ,0, 2 ) - >e pkoeronle_cla: c he _ i t em _ m a gi c _ ch e c k : m b u fp l c pu f r ee l i s t m o d if i e d : i t ema d dr 0 xpfagfef f fd 8 0 a 4 1 c3 f 0 0 +2 40 x a f5 5 1 e 6f 8 f 9 0 25 5 f != 0 x a f 55 1 e 6 f8 f 35 f d 5 f fStopped at db_enter+0x10: popq%rbp TIDPIDUID PRFLAGS PFLAGS CPU COMMAND 317552 39553 0 0x14000 0x2002K softnet 504828 12606 0 0x14000 0x2004 softnet *283345 81494 0 0x14000 0x2003 softnet db_enter() at db_enter+0x10 panic(81f39222) at panic+0xbf pool_cache_get(82323228) at pool_cache_get+0x25b pool_get(82323228,2) at pool_get+0x61 m_gethdr(2,1) at m_gethdr+0x3f pfsync_sendout() at pfsync_sendout+0xe9 pfsync_update_state(fd839f1f8950) at pfsync_update_state+0x15b pf_test(18,1,80095048,800022c6ae30) at pf_test+0xd53 veb_pf(80095048,1,fd80a3594900) at veb_pf+0xbf veb_port_input(80095048,fd80a3594900,ecf4bbdaf7f8,80747300) at veb_port_input+0x2ce ether_input(80095048,fd80a3594900) at ether_input+0x100 if_input_process(80095048,800022c6afc8) at if_input_process+0x6f ifiq_process(80099800) at ifiq_process+0x69 taskq_thread(80031100) at taskq_thread+0x11a end trace frame: 0x0, count: 1 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{3}> show panic *cpu3: pool_cache_item_magic_check: mbufpl cpu free list modified: item addr 0xfd80a41c3f00+24 0xaf551e6f8f90255f!=0xaf551e6f8f35fd5f cpu2: uvm_fault(0x823ba618, 0x17, 0, 2) -> e ddb{3}> ddb{3}> show reg rdi0 rsi 0x14 rbp 0x800022c6a960 rbx 0xfd842f835c00 rdx 0xc800 rcx0x282 rax 0x8a r8 0x101010101010101 r9 0 r10 0xedcd3183c339b665 r110xb0f0eb58b1d2563 r12 0x80002241ca60 r130 r140 r15 0x81f39222cmd0646_9_tim_udma+0x314d8 rip 0x8118e200db_enter+0x10 cs 0x8 rflags 0x206 rsp 0x800022c6a960 ss 0x10 db_enter+0x10: popq%rbp ddb{3}> show all locks Process 39553 (softnet) thread 0x8000ed20 (317552) shared rwlock netlock r = 0 (0x822c6550) shared rwlock softnet r = 0 (0x80031370) Process 12606 (softnet) thread 0x8000e000 (504828) shared rwlock netlock r = 0 (0x822c6550) shared rwlock softnet r = 0 (0x80031270) Process 81494 (softnet) thread 0x8000e2a0 (283345) shared rwlock netlock r = 0 (0x822c6550) shared rwlock softnet r = 0 (0x80031170) Process 96881 (softnet) thread 0x8000e540 (159803) shared rwlock softnet r = 0 (0x80031070) Process 26865 (systq) thread 0x8000ea80 (449324) shared rwlock systq r = 0 (0x822dd728) Process 93339 (softclock) thread 0x8000efc0 (160018) shared rwlock timeout r = 0 (0x822b6000) ddb{3}> ddb{3}> ps PID TID PPIDUID S FLAGS WAIT COMMAND 39455 263128 42512 0 3 0x3 netlock ifconfig 42512 258140 1 0 30x10008b sigsusp ksh 34696 282706 1 0 30x100098 kqreadcron 86943 298932 81565 95 3 0x1100092 kqreadsmtpd 34037 448643 81565103 3 0x1100092 kqreadsmtpd 17802 340759 81565 95 3 0x1100092 kqreadsmtpd 54979 438478 81565 95 30x100092 kqreadsmtpd 29724 438684 81565 95 3 0x1100092 kqreadsmtpd 3110 313509 81565 95 3 0x1100092 kqreadsmtpd 81565 137591 1 0 30x100080 kqreadsmtpd 81008 204817 1 0 30x88 kqreadsshd 72442 275002 1 0 30x100080 kqreadntpd 97406 453489 91190 83 30x100092 kqreadntpd 91190 488051 1 83 3 0x1100012 netlock ntpd 31521 42595 4468 73 3 0x1100090 kqreadsyslogd 4468 43476 1 0 30x100082 netio syslogd 66713 499933 0 0 3 0x14200 bored smr 73