Re: dwqe ifconfig down panic
On Fri, Mar 01, 2024 at 12:00:29AM +0100, Alexander Bluhm wrote:
> Hi,
>
> When doing flood ping transmit from a machine and simultaneously
> ifconfig down/up in a loop, dwqe(4) interface driver crashes.
>
> dwqe_down() contains an interrupt barrier, but somehow it does not
> work. Immediately after Xspllower() a transmit interrupt is
> processed.
>
> bluhm
Unfortunately I can't see it in the dmesg, but I wonder: Is it MSIs?
Maybe the edge-triggered interrupt stays in the controller because it
isn't cleared. But things you could try are:
* Clear the IRQ status in addition to disabling them. This might not
do something in case the MSI is already in the IRQ, there are no
takebacks. But then maybe when the interrupt fires, the code path
sees the cleared status and doesn't run the tx/rx proc.
* Don't run TX/RX proc in case the interface is down?
Cheers,
Patrick
> kernel: protection fault trap, code=0
> Stopped at m_tag_delete_chain+0x30:movq0(%rsi),%rax
>
> ddb{0}> trace
> m_tag_delete_chain(fd806bfa5300) at m_tag_delete_chain+0x30
> m_free(fd806bfa5300) at m_free+0x9e
> m_freem(fd806bfa5300) at m_freem+0x38
> dwqe_tx_proc(80304800) at dwqe_tx_proc+0x194
> dwqe_intr(80304800) at dwqe_intr+0x9b
> intr_handler(80003f86e760,805f4f80) at intr_handler+0x72
> Xintr_ioapic_edge36_untramp() at Xintr_ioapic_edge36_untramp+0x18f
> Xspllower() at Xspllower+0x1d
> dwqe_ioctl(80304870,80206910,80003f86e990) at dwqe_ioctl+0x18c
> ifioctl(fd81ffabe1e8,80206910,80003f86e990,80003f94e550) at
> ifioctl+0x726
> sys_ioctl(80003f94e550,80003f86eb50,80003f86eac0) at
> sys_ioctl+0x2af
> syscall(80003f86eb50) at syscall+0x55b
> Xsyscall() at Xsyscall+0x128
> end of kernel
> end trace frame: 0x73ef48509270, count: -13
>
> ddb{0}> show register
> rdi 0xfd806bfa5300
> rsi 0xdeafbeaddeafbead
> rbp 0x80003f86e5f0
> rbx0xf40
> rdx0
> rcx0
> rax 0xab56__ALIGN_SIZE+0x9b56
> r8 0x90
> r9 0x24634ac__kernel_rodata_phys+0x3624ac
> r10 0xe676ed611cc13e4f
> r11 0xd2619954b795f246
> r12 0x81110f48
> r13 0xfd807282
> r14 0xfd806bfa5300
> r15 0xfd805f6def00
> rip 0x81daae80m_tag_delete_chain+0x30
> cs 0x8
> rflags 0x10282__ALIGN_SIZE+0xf282
> rsp 0x80003f86e5d0
> ss 0x10
> m_tag_delete_chain+0x30:movq0(%rsi),%rax
>
> ddb{0}> x/s version
> version:OpenBSD 7.5 (GENERIC.MP) #2: Thu Feb 29 23:42:26 CET 2024\012
>r...@ot50.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP\012
>
> ddb{0}> ps
>PID TID PPIDUID S FLAGS WAIT COMMAND
> *70039 16536 80360 0 7 0x803ifconfig
> 41531 214934 36719 51 3 0x8100033 netlock ping
>
> OpenBSD 7.5 (GENERIC.MP) #2: Thu Feb 29 23:42:26 CET 2024
> r...@ot50.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 8038207488 (7665MB)
> avail mem = 7773556736 (7413MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.3 @ 0x769c7000 (85 entries)
> bios0: vendor American Megatrends Inc. version "1.02.10" date 06/27/2022
> efi0 at bios0: UEFI 2.7
> efi0: American Megatrends rev 0x50013
> acpi0 at bios0: ACPI 6.2
> acpi0: sleep states S0 S5
> acpi0: tablesfg0: addr 0xc000, bus 0-255
> acpihpet0 at acpi0: 1920 Hz
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel Atom(R) x6425RE Processor @ 1.90GHz, 1895.90 MHz, 06-96-01, patch
> 0017
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SMEP,ERMS,RDSEED,SMAP,CLFLUSHOPT,CLWB,PT,SHA,UMIP,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,MISC_PKG_CT,ENERGY_FILT,FB_CLEAR,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
> 12-way L2 cache, 4MB 64b/line 16-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 38MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.0.2.2.1.1.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel Atom(R) x6425RE Processor @ 1.90
Re: dwqe ifconfig down panic
> Date: Tue, 26 Mar 2024 23:05:49 +0100
> From: Patrick Wildt
>
> On Fri, Mar 01, 2024 at 12:00:29AM +0100, Alexander Bluhm wrote:
> > Hi,
> >
> > When doing flood ping transmit from a machine and simultaneously
> > ifconfig down/up in a loop, dwqe(4) interface driver crashes.
> >
> > dwqe_down() contains an interrupt barrier, but somehow it does not
> > work. Immediately after Xspllower() a transmit interrupt is
> > processed.
> >
> > bluhm
>
> Unfortunately I can't see it in the dmesg, but I wonder: Is it MSIs?
> Maybe the edge-triggered interrupt stays in the controller because it
> isn't cleared. But things you could try are:
>
> * Clear the IRQ status in addition to disabling them. This might not
> do something in case the MSI is already in the IRQ, there are no
> takebacks. But then maybe when the interrupt fires, the code path
> sees the cleared status and doesn't run the tx/rx proc.
> * Don't run TX/RX proc in case the interface is down?
Another thing... Is that intr_barrier() called while we're at
IPL_NET? That might not have the desired effect if intr_barrier()
runs on the same CPU that is handling the interrupts for the device.
And I fear that would be an issue in other drivers too...
> > kernel: protection fault trap, code=0
> > Stopped at m_tag_delete_chain+0x30:movq0(%rsi),%rax
> >
> > ddb{0}> trace
> > m_tag_delete_chain(fd806bfa5300) at m_tag_delete_chain+0x30
> > m_free(fd806bfa5300) at m_free+0x9e
> > m_freem(fd806bfa5300) at m_freem+0x38
> > dwqe_tx_proc(80304800) at dwqe_tx_proc+0x194
> > dwqe_intr(80304800) at dwqe_intr+0x9b
> > intr_handler(80003f86e760,805f4f80) at intr_handler+0x72
> > Xintr_ioapic_edge36_untramp() at Xintr_ioapic_edge36_untramp+0x18f
> > Xspllower() at Xspllower+0x1d
> > dwqe_ioctl(80304870,80206910,80003f86e990) at dwqe_ioctl+0x18c
> > ifioctl(fd81ffabe1e8,80206910,80003f86e990,80003f94e550) at
> > ifioctl+0x726
> > sys_ioctl(80003f94e550,80003f86eb50,80003f86eac0) at
> > sys_ioctl+0x2af
> > syscall(80003f86eb50) at syscall+0x55b
> > Xsyscall() at Xsyscall+0x128
> > end of kernel
> > end trace frame: 0x73ef48509270, count: -13
> >
> > ddb{0}> show register
> > rdi 0xfd806bfa5300
> > rsi 0xdeafbeaddeafbead
> > rbp 0x80003f86e5f0
> > rbx0xf40
> > rdx0
> > rcx0
> > rax 0xab56__ALIGN_SIZE+0x9b56
> > r8 0x90
> > r9 0x24634ac__kernel_rodata_phys+0x3624ac
> > r10 0xe676ed611cc13e4f
> > r11 0xd2619954b795f246
> > r12 0x81110f48
> > r13 0xfd807282
> > r14 0xfd806bfa5300
> > r15 0xfd805f6def00
> > rip 0x81daae80m_tag_delete_chain+0x30
> > cs 0x8
> > rflags 0x10282__ALIGN_SIZE+0xf282
> > rsp 0x80003f86e5d0
> > ss 0x10
> > m_tag_delete_chain+0x30:movq0(%rsi),%rax
> >
> > ddb{0}> x/s version
> > version:OpenBSD 7.5 (GENERIC.MP) #2: Thu Feb 29 23:42:26 CET
> > 2024\012
> > r...@ot50.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP\012
> >
> > ddb{0}> ps
> >PID TID PPIDUID S FLAGS WAIT COMMAND
> > *70039 16536 80360 0 7 0x803ifconfig
> > 41531 214934 36719 51 3 0x8100033 netlock ping
> >
> > OpenBSD 7.5 (GENERIC.MP) #2: Thu Feb 29 23:42:26 CET 2024
> > r...@ot50.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > real mem = 8038207488 (7665MB)
> > avail mem = 7773556736 (7413MB)
> > random: good seed from bootblocks
> > mpath0 at root
> > scsibus0 at mpath0: 256 targets
> > mainbus0 at root
> > bios0 at mainbus0: SMBIOS rev. 3.3 @ 0x769c7000 (85 entries)
> > bios0: vendor American Megatrends Inc. version "1.02.10" date 06/27/2022
> > efi0 at bios0: UEFI 2.7
> > efi0: American Megatrends rev 0x50013
> > acpi0 at bios0: ACPI 6.2
> > acpi0: sleep states S0 S5
> > acpi0: tablesfg0: addr 0xc000, bus 0-255
> > acpihpet0 at acpi0: 1920 Hz
> > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: Intel Atom(R) x6425RE Processor @ 1.90GHz, 1895.90 MHz, 06-96-01,
> > patch 0017
> > cpu0:
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SMEP,ERMS,RDSEED,SMAP,CLFLUSHOPT,CLWB,PT,SHA,UMIP,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,IBRS_ALL,SKIP_L1DF
Re: FW: ls -l Segmentation fault
Le Tue, Mar 26, 2024 at 01:58:37PM -0600, Todd C. Miller a écrit :
> On Tue, 26 Mar 2024 12:45:09 -0700, guent...@openbsd.org wrote:
>
> > Someone want to craft a diff for ls to handle that (and scan the tree for
> > other unchecked localtime() calls)? Not sure if
> > POSIX's ls spec has an out for how to print the time for such a thing.
>
> Another option is to just use the epoch for invalid timestamps.
> POSIX doesn't appear to offer guidance in this. NetBSD prints a
> string of '?' and FreeBSD prints "bad date val".
>
> - todd
>
OK denis@
> Index: bin/ls/print.c
> ===
> RCS file: /cvs/src/bin/ls/print.c,v
> retrieving revision 1.40
> diff -u -p -u -r1.40 print.c
> --- bin/ls/print.c7 Oct 2023 11:51:08 - 1.40
> +++ bin/ls/print.c26 Mar 2024 19:54:54 -
> @@ -241,6 +241,7 @@ static void
> printtime(time_t ftime)
> {
> char f_date[DATELEN];
> + struct tm *tm;
> static time_t now;
> static int now_set = 0;
>
> @@ -252,9 +253,14 @@ printtime(time_t ftime)
> /*
>* convert time to string, and print
>*/
> + if ((tm = localtime(&ftime)) == NULL) {
> + /* Invalid time stamp, just display the epoch. */
> + ftime = 0;
> + tm = localtime(&ftime);
> + }
> if (strftime(f_date, sizeof(f_date), f_sectime ? "%b %e %H:%M:%S %Y" :
> (ftime <= now - SIXMONTHS || ftime > now) ? "%b %e %Y" :
> - "%b %e %H:%M", localtime(&ftime)) == 0)
> + "%b %e %H:%M", tm) == 0)
> f_date[0] = '\0';
>
> printf("%s ", f_date);
>
Re: dwqe ifconfig down panic
On Tue, Mar 26, 2024 at 11:05:49PM +0100, Patrick Wildt wrote: > On Fri, Mar 01, 2024 at 12:00:29AM +0100, Alexander Bluhm wrote: > > Hi, > > > > When doing flood ping transmit from a machine and simultaneously > > ifconfig down/up in a loop, dwqe(4) interface driver crashes. > * Don't run TX/RX proc in case the interface is down? The RX path already has a corresponding check. But the Tx path does not. If the problem is a race involving mbufs freed via dwqe_down() and mbufs freed via dwqe_tx_proc() then this simple tweak might help. diff /usr/src commit - 029d0a842cd8a317375b31145383409491d345e7 path + /usr/src blob - 97f874d2edf74a009a811455fbf37ca56f725eef file + sys/dev/ic/dwqe.c --- sys/dev/ic/dwqe.c +++ sys/dev/ic/dwqe.c @@ -593,6 +593,9 @@ dwqe_tx_proc(struct dwqe_softc *sc) struct dwqe_buf *txb; int idx, txfree; + if ((ifp->if_flags & IFF_RUNNING) == 0) + return; + bus_dmamap_sync(sc->sc_dmat, DWQE_DMA_MAP(sc->sc_txring), 0, DWQE_DMA_LEN(sc->sc_txring), BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); >
Re: panic: "wakeup: p_stat is 2" using btrace(8) & vmd(8)
On Sun, Mar 24, 2024 at 11:03:24AM +0100, Martin Pieuchot wrote:
> On 22/02/24(Thu) 17:24, Claudio Jeker wrote:
> > On Thu, Feb 22, 2024 at 04:16:57PM +0100, Martin Pieuchot wrote:
> > > On 21/02/24(Wed) 13:05, Claudio Jeker wrote:
> > > > On Tue, Feb 20, 2024 at 09:34:12PM +0100, Martin Pieuchot wrote:
> > > > > On 28/10/21(Thu) 05:45, Visa Hankala wrote:
> > > > > > On Wed, Oct 27, 2021 at 09:02:08PM -0400, Dave Voutila wrote:
> > > > > > > Dave Voutila writes:
> > > > > > >
> > > > > > > > Was tinkering on a bt(5) script for trying to debug an issue in
> > > > > > > > vmm(4)
> > > > > > > > when I managed to start hitting a panic "wakeup: p_stat is 2"
> > > > > > > > being
> > > > > > > > triggered by kqueue coming from the softnet kernel task.
> > > > > > > >
> > > > > > > > I'm running an amd64 kernel built from the tree today (latest
> > > > > > > > CVS commit
> > > > > > > > id UynQo1r7kLKA0Q2p) with VMM_DEBUG option set and the defaults
> > > > > > > > from
> > > > > > > > GENERIC.MP. Userland is from the latest amd snap.
> > > > > > > >
> > > > > > > > To reproduce, I'm running a single OpenBSD-current guest under
> > > > > > > > vmd(8)
> > > > > > > > which I'm targeting with the following trivial btrace script I
> > > > > > > > was
> > > > > > > > working on to use for debugging something in vmm(4):
> > > > > > > >
> > > > > > > > tracepoint:sched:sleep / pid == $1 && tid == $2 /{
> > > > > > > > printf("pid %d, tid %d slept %d!\n", pid, tid, nsecs);
> > > > > > > > }
> > > > > > > >
> > > > > > > > tracepoint:sched:wakeup / pid == $1 && tid == $2 /{
> > > > > > > > printf("pid %d, tid %d awoke %d!\n", pid, tid, nsecs);
> > > > > > > > }
> > > > > > >
> > > > > > > Even easier reproduction: if you have 2 machines and can use
> > > > > > > tcpbench(1)
> > > > > > > between them, then while tcpbench is running target it with the
> > > > > > > above
> > > > > > > btrace script. I've found running the script, killing it with
> > > > > > > ctrl-c,
> > > > > > > and re-running it 2-3 times triggers the panic on my laptop.
> > > > > > >
> > > > > > > >
> > > > > > > > Both times this happened I was trying to sysupgrade the vmd(8)
> > > > > > > > guest
> > > > > > > > while running the above btrace script. When I don't run the
> > > > > > > > script,
> > > > > > > > there is no panic.
> > > > > > > >
> > > > > > > > Image of the full backtrace is here: https://imgur.com/a/swW1qoj
> > > > > > > >
> > > > > > > > Simple transcript of the call stack after the panic() call
> > > > > > > > looks like:
> > > > > > > >
> > > > > > > > wakeup_n
> > > > > > > > kqueue_wakeup
> > > > > > > > knote
> > > > > > > > selwakekup
> > > > > > > > tun_enqueue
> > > > > > > > ether_output
> > > > > > > > ip_output
> > > > > > > > ip_forward
> > > > > > > > ip_input_if
> > > > > > > > ipv4_input
> > > > > > > > ether_input
> > > > > > > > if_input_process
> > > > > > > >
> > > > > > > > The other 3 cpu cores appeared to be in ipi handlers. (Image in
> > > > > > > > that
> > > > > > > > imgur link)
> > > > > >
> > > > > > I think the problem is recursion within the scheduler. Trace points
> > > > > > invoke wakeup() directly, which is unsafe when done from within the
> > > > > > run queue routines.
> > > > > >
> > > > > > One approach to avoid the recursion is to defer the wakeup() with
> > > > > > a soft interrupt. The scheduler runs at an elevated system priority
> > > > > > level that blocks soft interrupts. The deferred wakeup() is issued
> > > > > > once
> > > > > > the system priority level turns low enough.
> > > > > >
> > > > > > Unfortunately, the patch will get broken when someone adds trace
> > > > > > points
> > > > > > to soft interrupt scheduling...
> > > > >
> > > > > Thanks for the report. Sorry for the delay. I'm now really
> > > > > interested
> > > > > in fixing this bug because I'm heavily using btrace(8) to analyse the
> > > > > scheduler and I hit this panic a couple of times.
> > > > >
> > > > > The problem is that `pnext' might no longer be on the sleepqueue
> > > > > after a
> > > > > tracepoint inside setrunnable() fired. Diff below fixes that by
> > > > > making
> > > > > wakeup_n() re-entrant.
> > > > >
> > > > > I'm not very interested in committing this diff because it relies on a
> > > > > recursive SCHED_LOCK(). Instead I'd prefer to split wakeup_n() in two
> > > > > stages: first unlink the threads then call setrunnable(). This
> > > > > approach
> > > > > will help us untangle the sleepqueue but needs a bit more shuffling,
> > > > > like moving unsleep() out of setrunnable()...
> > > > >
> > > > > Claudio, Visa do you agree?
> > > >
> > > > [...]
> > > >
> > > > Ugh, this is too much magic for my little brain.
> > > > I would prefer to unqueue the procs onto a new local list and have a
> > > > special wakeup_proc for them. The code around wakeup and unsleep will
> > > > need
> > > > some changes to unlock it properly.
> > >
> > > Yes, that is what I
