Hi Roland, Pretty late indeed, but here are two SRIOV locking related fixes for issues spotted by lockdep. Basically, as we're late in the cycle, I guess there are two options here, either push to 3.8 and later to -stable once 3.7 is released, or push to 3.7 this week. This way or another, would be nice if you can place them on a branch at your tree which will cause them to spend the night in linux-next.
Or. Jack Morgenstein (2): IB/mlx4: Fix spinlock order to avoid lockdep warnings NET/mlx4_core: Fix potential deadlock in mlx4_eq_int drivers/infiniband/hw/mlx4/cm.c | 4 ++-- drivers/net/ethernet/mellanox/mlx4/cmd.c | 9 +++++---- drivers/net/ethernet/mellanox/mlx4/eq.c | 10 ++++++---- 3 files changed, 13 insertions(+), 10 deletions(-) These are the lockdep warnings, initially those that relate to patch #1 and later those relating to patch #2. Personally, I don't see the added value in placing them in the change-log, so they are brought up here for the purpose of reporting on the problems. (1) those that relate to patch #1 ====================================================== [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ] 3.7.0-rc6+ #68 Not tainted ------------------------------------------------------ kworker/u:3/1547 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: (&(&dev->sriov.id_map_lock)->rlock){+.+...}, at: [<ffffffffa03d5387>] schedule_delayed+0x38/0x77 [mlx4_ib] and this task is already holding: (&(&dev->sriov.going_down_lock)->rlock){-.-...}, at: [<ffffffffa03d537c>] schedule_delayed+0x2d/0x77 [mlx4_ib] which would create a new lock dependency: (&(&dev->sriov.going_down_lock)->rlock){-.-...} -> (&(&dev->sriov.id_map_lock)->rlock){+.+...} but this new dependency connects a HARDIRQ-irq-safe lock: (&(&dev->sriov.going_down_lock)->rlock){-.-...} ... which became HARDIRQ-irq-safe at: [<ffffffff81070ea6>] __lock_acquire+0x5e9/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813845e8>] _raw_spin_lock_irqsave+0x46/0x58 [<ffffffffa03c8ad9>] mlx4_ib_tunnel_comp_handler+0x22/0x57 [mlx4_ib] [<ffffffffa03c63ba>] mlx4_ib_cq_comp+0x12/0x14 [mlx4_ib] [<ffffffffa03e9b22>] mlx4_cq_completion+0x5c/0x62 [mlx4_core] [<ffffffffa03eb1fb>] mlx4_eq_int+0x89/0x865 [mlx4_core] [<ffffffffa03eb9e6>] mlx4_msi_x_interrupt+0xf/0x16 [mlx4_core] [<ffffffff8108b0d7>] handle_irq_event_percpu+0x93/0x1d7 [<ffffffff8108b257>] handle_irq_event+0x3c/0x5c [<ffffffff8108ddad>] handle_edge_irq+0xcc/0xf3 [<ffffffff810032e7>] handle_irq+0x1f/0x28 [<ffffffff81002a94>] do_IRQ+0x48/0xaf [<ffffffff81384fef>] ret_from_intr+0x0/0x13 [<ffffffff81008aa9>] cpu_idle+0x6e/0xab [<ffffffff8137d16a>] start_secondary+0x1af/0x1b3 to a HARDIRQ-irq-unsafe lock: (&(&dev->sriov.id_map_lock)->rlock){+.+...} ... which became HARDIRQ-irq-unsafe at: ... [<ffffffff81070f1f>] __lock_acquire+0x662/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d55bd>] mlx4_ib_multiplex_cm_handler+0x128/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&(&dev->sriov.id_map_lock)->rlock); local_irq_disable(); lock(&(&dev->sriov.going_down_lock)->rlock); lock(&(&dev->sriov.id_map_lock)->rlock); <Interrupt> lock(&(&dev->sriov.going_down_lock)->rlock); *** DEADLOCK *** 3 locks held by kworker/u:3/1547: #0: ((name)#2){.+.+.+}, at: [<ffffffff81045fa3>] process_one_work+0x238/0x498 #1: ((&ctx->work)#2){+.+.+.}, at: [<ffffffff81045fa3>] process_one_work+0x238/0x498 #2: (&(&dev->sriov.going_down_lock)->rlock){-.-...}, at: [<ffffffffa03d537c>] schedule_delayed+0x2d/0x77 [mlx4_ib] the dependencies between HARDIRQ-irq-safe lock and the holding lock: -> (&(&dev->sriov.going_down_lock)->rlock){-.-...} ops: 1222 { IN-HARDIRQ-W at: [<ffffffff81070ea6>] __lock_acquire+0x5e9/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813845e8>] _raw_spin_lock_irqsave+0x46/0x58 [<ffffffffa03c8ad9>] mlx4_ib_tunnel_comp_handler+0x22/0x57 [mlx4_ib] [<ffffffffa03c63ba>] mlx4_ib_cq_comp+0x12/0x14 [mlx4_ib] [<ffffffffa03e9b22>] mlx4_cq_completion+0x5c/0x62 [mlx4_core] [<ffffffffa03eb1fb>] mlx4_eq_int+0x89/0x865 [mlx4_core] [<ffffffffa03eb9e6>] mlx4_msi_x_interrupt+0xf/0x16 [mlx4_core] [<ffffffff8108b0d7>] handle_irq_event_percpu+0x93/0x1d7 [<ffffffff8108b257>] handle_irq_event+0x3c/0x5c [<ffffffff8108ddad>] handle_edge_irq+0xcc/0xf3 [<ffffffff810032e7>] handle_irq+0x1f/0x28 [<ffffffff81002a94>] do_IRQ+0x48/0xaf [<ffffffff81384fef>] ret_from_intr+0x0/0x13 [<ffffffff81008aa9>] cpu_idle+0x6e/0xab [<ffffffff8137d16a>] start_secondary+0x1af/0x1b3 IN-SOFTIRQ-W at: [<ffffffff81070ec9>] __lock_acquire+0x60c/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813845e8>] _raw_spin_lock_irqsave+0x46/0x58 [<ffffffffa03c8ad9>] mlx4_ib_tunnel_comp_handler+0x22/0x57 [mlx4_ib] [<ffffffffa03c63ba>] mlx4_ib_cq_comp+0x12/0x14 [mlx4_ib] [<ffffffffa03e9b22>] mlx4_cq_completion+0x5c/0x62 [mlx4_core] [<ffffffffa03eb1fb>] mlx4_eq_int+0x89/0x865 [mlx4_core] [<ffffffffa03eb9e6>] mlx4_msi_x_interrupt+0xf/0x16 [mlx4_core] [<ffffffff8108b0d7>] handle_irq_event_percpu+0x93/0x1d7 [<ffffffff8108b257>] handle_irq_event+0x3c/0x5c [<ffffffff8108ddad>] handle_edge_irq+0xcc/0xf3 [<ffffffff810032e7>] handle_irq+0x1f/0x28 [<ffffffff81002a94>] do_IRQ+0x48/0xaf [<ffffffff81384fef>] ret_from_intr+0x0/0x13 [<ffffffff812cd807>] flush_unmaps_timeout+0x2c/0x31 [<ffffffff8103be30>] call_timer_fn+0xb2/0x178 [<ffffffff8103c0ed>] run_timer_softirq+0x1f7/0x24d [<ffffffff81035d86>] __do_softirq+0x10c/0x1fe [<ffffffff8138cbcc>] call_softirq+0x1c/0x30 [<ffffffff81003280>] do_softirq+0x38/0x80 [<ffffffff81035a3c>] irq_exit+0x4e/0x83 [<ffffffff8101e8d7>] smp_apic_timer_interrupt+0x86/0x94 [<ffffffff8138c52f>] apic_timer_interrupt+0x6f/0x80 [<ffffffff81008aa9>] cpu_idle+0x6e/0xab [<ffffffff8137d16a>] start_secondary+0x1af/0x1b3 INITIAL USE at: [<ffffffff81070f9a>] __lock_acquire+0x6dd/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813845e8>] _raw_spin_lock_irqsave+0x46/0x58 [<ffffffffa03caeb9>] do_slave_init+0x164/0x1be [mlx4_ib] [<ffffffffa03cbd85>] mlx4_ib_add+0x966/0xa43 [mlx4_ib] [<ffffffffa03eefed>] mlx4_add_device+0x47/0x9a [mlx4_core] [<ffffffffa03ef127>] mlx4_register_interface+0x5a/0x93 [mlx4_core] [<ffffffffa03df053>] 0xffffffffa03df053 [<ffffffff810001fa>] do_one_initcall+0x7a/0x12e [<ffffffff8107beed>] sys_init_module+0x7a/0x1bd [<ffffffff8138ba12>] system_call_fastpath+0x16/0x1b } ... key at: [<ffffffffa03da250>] __key.29772+0x0/0xffffffffffffd7cf [mlx4_ib] ... acquired at: [<ffffffff8106f0f6>] check_irq_usage+0x5d/0xbe [<ffffffff8106fe59>] validate_chain+0x890/0xe5a [<ffffffff810711f7>] __lock_acquire+0x93a/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d5387>] schedule_delayed+0x38/0x77 [mlx4_ib] [<ffffffffa03d576f>] mlx4_ib_multiplex_cm_handler+0x2da/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 the dependencies between the lock to be acquired and HARDIRQ-irq-unsafe lock: -> (&(&dev->sriov.id_map_lock)->rlock){+.+...} ops: 5 { HARDIRQ-ON-W at: [<ffffffff81070f1f>] __lock_acquire+0x662/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d55bd>] mlx4_ib_multiplex_cm_handler+0x128/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 SOFTIRQ-ON-W at: [<ffffffff81070f42>] __lock_acquire+0x685/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d55bd>] mlx4_ib_multiplex_cm_handler+0x128/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 INITIAL USE at: [<ffffffff81070f9a>] __lock_acquire+0x6dd/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d55bd>] mlx4_ib_multiplex_cm_handler+0x128/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 } ... key at: [<ffffffffa03da378>] __key.27645+0x0/0xffffffffffffd6a7 [mlx4_ib] ... acquired at: [<ffffffff8106f0f6>] check_irq_usage+0x5d/0xbe [<ffffffff8106fe59>] validate_chain+0x890/0xe5a [<ffffffff810711f7>] __lock_acquire+0x93a/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d5387>] schedule_delayed+0x38/0x77 [mlx4_ib] [<ffffffffa03d576f>] mlx4_ib_multiplex_cm_handler+0x2da/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 stack backtrace: Pid: 1547, comm: kworker/u:3 Not tainted 3.7.0-rc6+ #68 Call Trace: [<ffffffff81030325>] ? console_unlock+0x358/0x37e [<ffffffff8106f085>] check_usage+0x525/0x539 [<ffffffff8106f0f6>] check_irq_usage+0x5d/0xbe [<ffffffff8106fe59>] validate_chain+0x890/0xe5a [<ffffffff8106e463>] ? trace_hardirqs_on+0xd/0xf [<ffffffff810711f7>] __lock_acquire+0x93a/0x9ae [<ffffffff8107135b>] lock_acquire+0xf0/0x116 [<ffffffffa03d5387>] ? schedule_delayed+0x38/0x77 [mlx4_ib] [<ffffffff813844e5>] _raw_spin_lock+0x3b/0x4a [<ffffffffa03d5387>] ? schedule_delayed+0x38/0x77 [mlx4_ib] [<ffffffffa03d5387>] schedule_delayed+0x38/0x77 [mlx4_ib] [<ffffffffa03d576f>] mlx4_ib_multiplex_cm_handler+0x2da/0x2ff [mlx4_ib] [<ffffffffa03c900f>] mlx4_ib_multiplex_mad+0x1a4/0x2bd [mlx4_ib] [<ffffffff8106e41f>] ? trace_hardirqs_on_caller+0x11e/0x155 [<ffffffffa03c6c0e>] ? mlx4_ib_poll_cq+0x609/0x62d [mlx4_ib] [<ffffffffa03c9196>] mlx4_ib_tunnel_comp_worker+0x6e/0x140 [mlx4_ib] [<ffffffff8104604c>] process_one_work+0x2e1/0x498 [<ffffffff81045fa3>] ? process_one_work+0x238/0x498 [<ffffffff81384c23>] ? _raw_spin_unlock_irq+0x2b/0x40 [<ffffffffa03c9128>] ? mlx4_ib_multiplex_mad+0x2bd/0x2bd [mlx4_ib] [<ffffffff8104669d>] worker_thread+0x225/0x35d [<ffffffff81046478>] ? manage_workers+0x275/0x275 [<ffffffff8104c887>] kthread+0xc2/0xca [<ffffffff8104c7c5>] ? __init_kthread_worker+0x56/0x56 [<ffffffff8138b96c>] ret_from_fork+0x7c/0xb0 [<ffffffff8104c7c5>] ? __init_kthread_worker+0x56/0x56 [sched_delayed] sched: RT throttling activated (2) those that relate to patch #2 inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/7/0 [HC1[1]:SC0[0]:HE0:SE1] takes: (&(&priv->mfunc.master.slave_state_lock)->rlock){?.+...}, at: [<ffffffffa042e786>] mlx4_eq_int+0x5e7/0x84b [mlx4_core] {HARDIRQ-ON-W} state was registered at: [<ffffffff8107143f>] __lock_acquire+0x662/0x9ae [<ffffffff8107187b>] lock_acquire+0xf0/0x116 [<ffffffff813848ad>] _raw_spin_lock+0x3b/0x4a [<ffffffffa042c1d4>] mlx4_master_comm_channel+0x3e3/0x4d5 [mlx4_core] [<ffffffff81045f9c>] process_one_work+0x2e1/0x498 [<ffffffff810465ed>] worker_thread+0x225/0x35d [<ffffffff8104c7cb>] kthread+0xc2/0xca [<ffffffff8138bdec>] ret_from_fork+0x7c/0xb0 irq event stamp: 1140398 hardirqs last enabled at (1140395): [<ffffffff810086db>] mwait_idle+0x133/0x208 hardirqs last disabled at (1140396): [<ffffffff813853aa>] common_interrupt+0x6a/0x6f softirqs last enabled at (1140398): [<ffffffff810359f7>] _local_bh_enable+0xe/0x10 softirqs last disabled at (1140397): [<ffffffff81035bc0>] irq_enter+0x44/0x76 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&(&priv->mfunc.master.slave_state_lock)->rlock); <Interrupt> lock(&(&priv->mfunc.master.slave_state_lock)->rlock); *** DEADLOCK *** no locks held by swapper/7/0. stack backtrace: Pid: 0, comm: swapper/7 Not tainted 3.7.0-rc1+ #2 Call Trace: <IRQ> [<ffffffff8103021a>] ? console_unlock+0x2d5/0x37e [<ffffffff8106e086>] print_usage_bug+0x297/0x2a8 [<ffffffff8100cc92>] ? save_stack_trace+0x2a/0x47 [<ffffffff8106eee8>] ? print_irq_inversion_bug+0x1d7/0x1d7 [<ffffffff8106e38a>] mark_lock+0x2f3/0x52b [<ffffffff810713c6>] __lock_acquire+0x5e9/0x9ae [<ffffffff8107187b>] lock_acquire+0xf0/0x116 [<ffffffffa042e786>] ? mlx4_eq_int+0x5e7/0x84b [mlx4_core] [<ffffffff813848ad>] _raw_spin_lock+0x3b/0x4a [<ffffffffa042e786>] ? mlx4_eq_int+0x5e7/0x84b [mlx4_core] [<ffffffff8106e925>] ? trace_hardirqs_on_caller+0x104/0x155 [<ffffffffa042e786>] mlx4_eq_int+0x5e7/0x84b [mlx4_core] [<ffffffff8106e983>] ? trace_hardirqs_on+0xd/0xf [<ffffffffa042e9f9>] mlx4_msi_x_interrupt+0xf/0x16 [mlx4_core] [<ffffffff8108b5f3>] handle_irq_event_percpu+0x93/0x1d7 [<ffffffff8108b773>] handle_irq_event+0x3c/0x5c [<ffffffff8108e2c9>] handle_edge_irq+0xcc/0xf3 [<ffffffff81003407>] handle_irq+0x1f/0x28 [<ffffffff81002bb4>] do_IRQ+0x48/0xaf [<ffffffff813853af>] common_interrupt+0x6f/0x6f <EOI> [<ffffffff810086e4>] ? mwait_idle+0x13c/0x208 [<ffffffff810086db>] ? mwait_idle+0x133/0x208 [<ffffffff81008bc9>] cpu_idle+0x6e/0xab [<ffffffff8137d512>] start_secondary+0x1af/0x1b3 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html