On Fri, Mar 04, 2011 at 11:40:00AM -0700, Williams, Mitch A wrote:
>>From: Brandeburg, Jesse
>[snip]
>>these messages are due to irqbalance moving the interrupt, combined with a
>>kernel bug that is not handling the per-cpu stuff correctly for the irq
>>
>>I can't seem to find the patch that is relevant in upstream that might
>>have fixed this but I'm pretty sure there is one.
>
>This sounds similar to a bug I fixed back in 2.6.21. It's possible that
>we just need a read flush in the migration code.
Hi Mitch,
thanks very much for looking into this! much more than I expected from
a drivers mailing list :-)
>Robin, would it possible for you to try this patch? If it fixes your problem,
>I'll push it upstream.
I gave it a go today, but sadly the kernel crashed part way through
boot. it was insmod'ing an infiniband mlx4 module which judging from
the trace looks like it was unhappy with the pci/msix change.
sorry for the delay in replying.
cheers,
robin
...
2011-03-24 15:13:40 Freeing unused kernel memory: 624k freed
2011-03-24 15:13:40 Write protecting the kernel read-only data: 4152k
2011-03-24 15:13:40 Running /init
2011-03-24 15:13:40 dca service started, version 1.12.1
2011-03-24 15:13:40 Intel(R) Gigabit Ethernet Network Driver - version 1.3.16-k2
2011-03-24 15:13:40 Copyright (c) 2007-2009 Intel Corporation.
2011-03-24 15:13:40 mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3
(January 19, 2011)
2011-03-24 15:13:40 mlx4_core: Initializing 0000:02:00.0
2011-03-24 15:13:40 mlx4_core 0000:02:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
2011-03-24 15:13:42 BUG: unable to handle kernel paging request at
0000000000004061
2011-03-24 15:13:42 IP: [<ffffffff81194573>] write_msi_msg_desc+0x5a/0x125
2011-03-24 15:13:42 PGD 37ad0f067 PUD 37acc7067 PMD 0
2011-03-24 15:13:42 Oops: 0000 [#1] SMP
2011-03-24 15:13:42 last sysfs file: /sys/class/usbmon/usbmon0/dev
2011-03-24 15:13:42 CPU 0
2011-03-24 15:13:42 Modules linked in: mlx4_core(+) ib_mad ib_core igb dca
2011-03-24 15:13:42 Pid: 808, comm: insmod Tainted: G W
2.6.32.34-1.8.5a #1 SUN BLADE X6275 SERVER MODULE
2011-03-24 15:13:42 RIP: 0010:[<ffffffff81194573>] [<ffffffff81194573>]
write_msi_msg_desc+0x5a/0x125
2011-03-24 15:13:42 RSP: 0018:ffff88067b76bba8 EFLAGS: 00010282
2011-03-24 15:13:42 RAX: 0000000000004061 RBX: ffff88067b76bc28 RCX:
0000000000000000
2011-03-24 15:13:42 RDX: 0000000000004061 RSI: ffff88067b76bc28 RDI:
ffff88037b64fcc0
2011-03-24 15:13:42 RBP: ffff88067b76bbe8 R08: 0000000000000010 R09:
ffff88037e73f288
2011-03-24 15:13:42 R10: 00000000ffffffff R11: 0000000000000046 R12:
ffff88037e677800
2011-03-24 15:13:42 R13: ffff88067b76bc28 R14: ffff88037b69a140 R15:
ffff88067b76bc28
2011-03-24 15:13:42 FS: 00007f564a5326f0(0000) GS:ffff880028200000(0000)
knlGS:0000000000000000
2011-03-24 15:13:42 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2011-03-24 15:13:42 CR2: 0000000000004061 CR3: 000000037adba000 CR4:
00000000000006f0
2011-03-24 15:13:42 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
2011-03-24 15:13:42 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
2011-03-24 15:13:42 Process insmod (pid: 808, threadinfo ffff88067b76a000, task
ffff88067e5f6cd0)
2011-03-24 15:13:42 Stack:
2011-03-24 15:13:42 ffff88037e73f280 000000007e73f280 0000000000000032
ffff88067b76bc28
2011-03-24 15:13:42 <0> 0000000000000032 ffff88037e677800 ffff88037b69a140
ffff88067b76bc28
2011-03-24 15:13:42 <0> ffff88067b76bc08 ffffffff8119465f ffff88037e677800
0000000000000000
2011-03-24 15:13:42 Call Trace:
2011-03-24 15:13:42 [<ffffffff8119465f>] write_msi_msg+0x21/0x26
2011-03-24 15:13:42 [<ffffffff8102284c>] arch_setup_msi_irqs+0x13f/0x205
2011-03-24 15:13:42 [<ffffffff81194f07>] pci_enable_msix+0x335/0x361
2011-03-24 15:13:42 [<ffffffffa00544a3>] __mlx4_init_one+0x4c4/0x901
[mlx4_core]
2011-03-24 15:13:42 [<ffffffffa005ad7d>] mlx4_init_one+0x41/0x46 [mlx4_core]
2011-03-24 15:13:42 [<ffffffff8118d7c1>] local_pci_probe+0x17/0x1b
2011-03-24 15:13:42 [<ffffffff8118e40f>] pci_device_probe+0xc5/0xf1
2011-03-24 15:13:42 [<ffffffff812178a9>] ? driver_sysfs_add+0x52/0x78
2011-03-24 15:13:42 [<ffffffff81217a04>] driver_probe_device+0xb7/0x13b
2011-03-24 15:13:42 [<ffffffff81217ae5>] __driver_attach+0x5d/0x80
2011-03-24 15:13:42 [<ffffffff81217a88>] ? __driver_attach+0x0/0x80
2011-03-24 15:13:42 [<ffffffff81217a88>] ? __driver_attach+0x0/0x80
2011-03-24 15:13:42 [<ffffffff81216dc1>] bus_for_each_dev+0x4e/0x7f
2011-03-24 15:13:42 [<ffffffff81217855>] driver_attach+0x21/0x23
2011-03-24 15:13:42 [<ffffffff8121743b>] bus_add_driver+0xf9/0x266
2011-03-24 15:13:42 [<ffffffff81217e8a>] driver_register+0xa3/0x11a
2011-03-24 15:13:42 [<ffffffffa0065031>] ? mlx4_init+0x0/0xab [mlx4_core]
2011-03-24 15:13:42 [<ffffffff8118e68b>] __pci_register_driver+0x55/0xb1
2011-03-24 15:13:42 [<ffffffffa0065031>] ? mlx4_init+0x0/0xab [mlx4_core]
2011-03-24 15:13:42 [<ffffffffa0065031>] ? mlx4_init+0x0/0xab [mlx4_core]
2011-03-24 15:13:42 [<ffffffffa006509e>] mlx4_init+0x6d/0xab [mlx4_core]
2011-03-24 15:13:42 [<ffffffff8100906a>] do_one_initcall+0x5f/0x14f
2011-03-24 15:13:42 [<ffffffff81077cb6>] sys_init_module+0xd0/0x22a
2011-03-24 15:13:42 [<ffffffff8100bb9b>] system_call_fastpath+0x16/0x1b
2011-03-24 15:13:42 Code: 06 01 74 2e 41 0f b7 46 02 8b 16 c1 e0 04 48 98 49 03
46 20 89 10 8b 4e 04 48 8d 50 04 89 0a 8b 56 08 48 83 c0 08 89 10 8b 46 08 <8b>
00 e9 a5 00 00 00 45 0f b6 7e 01 41 8b 74 24 38 41 8d 5f 02
2011-03-24 15:13:42 RIP [<ffffffff81194573>] write_msi_msg_desc+0x5a/0x125
2011-03-24 15:13:42 RSP <ffff88067b76bba8>
2011-03-24 15:13:42 CR2: 0000000000004061
2011-03-24 15:13:42 ---[ end trace 4eaa2a86a8e2da24 ]---
>Thanks,
>Mitch
>
>diff -urpN -X dontdiff a/drivers/pci/msi.c b/drivers/pci/msi.c
>--- a/drivers/pci/msi.c 2009-12-02 19:51:21.000000000 -0800
>+++ b/drivers/pci/msi.c 2011-03-04 08:55:45.380138841 -0800
>@@ -239,6 +239,7 @@ void write_msi_msg_desc(struct irq_desc
> writel(msg->address_lo, base + PCI_MSIX_ENTRY_LOWER_ADDR);
> writel(msg->address_hi, base + PCI_MSIX_ENTRY_UPPER_ADDR);
> writel(msg->data, base + PCI_MSIX_ENTRY_DATA);
>+ readl(msg->data); /* Flush write to device */
> } else {
> struct pci_dev *dev = entry->dev;
> int pos = entry->msi_attrib.pos;
>
>
>------------------------------------------------------------------------------
>What You Don't Know About Data Connectivity CAN Hurt You
>This paper provides an overview of data connectivity, details
>its effect on application quality, and explores various alternative
>solutions. http://p.sf.net/sfu/progress-d2d
>_______________________________________________
>E1000-devel mailing list
>E1000-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/e1000-devel
>To learn more about Intel® Ethernet, visit
>http://communities.intel.com/community/wired
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
