Hi Thomas,
On Tue, Jun 25, 2019 at 07:32:03PM +0800, Thomas Gleixner wrote:
> Rong,
>
> On Tue, 25 Jun 2019, Rong Chen wrote:
> > On 6/25/19 2:24 PM, Thomas Gleixner wrote:
> > > > On 6/22/19 3:08 AM, Thomas Gleixner wrote:
> > > > > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge
> > > > > > -smp
> > > > > > 2 -m
> > > > > > 2G
> > > > > I cannot reproduce that issue. What's the underlying hardware machine?
> > > > brand: Genuine Intel(R) CPU 000 @ 2.27GHz
> > > > model: Westmere-EX
> > > > memory: 256G
> > > > nr_node: 4
> > > > nr_cpu: 80
> > > Ok. I'll try to find something similar. Can please you rerun that test on
> > > that particular configuration with the updated branch?
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.x86/ipi
> > >
> > I have tested commit e0b179bc1a ("x86/apic/x2apic: Add conditional IPI
> > shorthands support"), the problem is still exist.
>
> the head of that branch is:
>
> 4f3f6d6a7f8e ("x86/apic/x2apic: Add conditional IPI shorthands support")
>
> This is WIP and force pushed. There are no incremental changes. Could you
> please check again?
Since you can't reproduce it yet, we've added some debug hook to get more
info, like dmesg below:
[ 288.865969] kvm-clock: cpu 0, msr 6d201001, secondary cpu clock
[ 288.866069] IRR[7]: 0x1000
[ 289.890274] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1502
setup_local_APIC+0x2d1/0x4f0
[ 289.911792] Modules linked in: locktorture torture sr_mod cdrom ata_generic
pata_acpi bochs_drm ppdev ttm ata_piix drm_kms_helper snd_pcm libata
syscopyarea sysfillrect snd_timer snd sysimgblt soundcore fb_sys_fops joydev
crc32c_intel drm serio_raw pcspkr parport_pc i2c_piix4 floppy parport
[ 289.959629] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
5.2.0-rc5-00493-g1965704 #1
[ 289.975340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
[ 289.990568] RIP: 0010:setup_local_APIC+0x2d1/0x4f0
[ 290.000361] Code: 7a 7d 01 85 c9 0f 84 a9 00 00 00 0f 31 c1 e1 0a 48 c1 e2
20 41 89 ce 4c 03 34 24 48 09 d0 49 29 c6 4d 85 f6 0f 8f a1 fe ff ff <0f> 0b 48
c7 c7 09 08 cc 8e 89 ea 44 89 ee bb 07 00 00 00 48 c7 c5
[ 290.035653] RSP: 0000:ffffffff8f003ee0 EFLAGS: 00010086
[ 290.044854] RAX: 0000009c7aa7aeb4 RBX: 0000000000000020 RCX: 000000008a001800
[ 290.057690] RDX: 0000009c00000000 RSI: 0000000000000020 RDI: ffffffff8f003ef0
[ 290.070711] RBP: 0000000000000000 R08: 0000000000000341 R09: 000000000000001d
[ 290.083680] R10: ffff943bcf49b740 R11: ffffffff8f003d90 R12: 00000000000000f0
[ 290.096414] R13: 0000000000001000 R14: fffffffffffc581b R15: 0000000000000000
[ 290.109369] FS: 0000000000000000(0000) GS:ffff943c32400000(0000)
knlGS:0000000000000000
[ 290.124123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 290.134558] CR2: 00000000f7ec7710 CR3: 000000006c60a000 CR4: 00000000000006f0
[ 290.147451] Call Trace:
[ 290.152154] apic_ap_setup+0xa/0x20
[ 290.158663] start_secondary+0x78/0x1e0
[ 290.165411] secondary_startup_64+0xb6/0xc0
[ 290.173158] ---[ end trace bfefeb8bcdb1b9e3 ]---
[ 290.182418] queued = 0x1000 acked = 0
[ 290.189159] IRR[7]: 0x1000
Which shows the IRR[7] was set 0x1000, IIUC, it means vector
0xec, which is for LAPIC timer, and ISRs are all 0 before and
after the loop.
The debug code is as follows:
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 197a150..38e0851 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1447,6 +1447,22 @@ static void apic_pending_intr_clear(void)
if (boot_cpu_has(X86_FEATURE_TSC))
tsc = rdtsc();
+
+ {
+ for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+ queued = apic_read(APIC_IRR + i*0x10);
+
+ if (queued)
+ printk("IRR[%d]: 0x%x\n", i, queued);
+ }
+
+ for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+ value = apic_read(APIC_ISR + i*0x10);
+ if (value)
+ printk("ISR[%d]: 0x%lx\n", i, value);
+ }
+ }
+
/*
* After a crash, we no longer service the interrupts and a pending
* interrupt from previous kernel might still have ISR bit set.
@@ -1484,6 +1500,25 @@ static void apic_pending_intr_clear(void)
}
} while (queued && max_loops > 0);
WARN_ON(max_loops <= 0);
+
+ if (max_loops <= 0) {
+ printk("queued = 0x%x acked = %d\n", queued, acked);
+
+ queued = 0;
+ for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+ queued = apic_read(APIC_IRR + i*0x10);
+
+ if (queued)
+ printk("IRR[%d]: 0x%x\n", i, queued);
+ }
+
+ for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+ value = apic_read(APIC_ISR + i*0x10);
+ if (value)
+ printk("ISR[%d]: 0x%lx\n", i, value);
+
+ }
+ }
}
/**
Thanks,
Feng
